AD-A204  922 


AFHRL-TP-88-39 


TTW  Cnr  r 


AIR  FORCE  m 

H 


EXPLORATORY  MODELS  TO  LINK  JOB 
PERFORMANCE  TO  ENLISTMENT  STANDARDS 


C.  Deene  Gott 

Advanced  Technology,  Incorporated 
461 S  Ualzea  Road 
Suite  208 

San  Antonio,  Texas  78218 

Under  Subcontract  To: 

Universal  Energy  Systeas,  Incorporated 
Manpower,  Personnel,  and  Training  Division 
4401  Dayton-Xenia  Road 
Dayton,  Ohio  45432-1894 


TRAINING  SYSTEMS  DIVISION 
Brooks  Air  Force  Base,  Texas  78235-5601 


December  1968 

Final  Technical  Paper  for  Period  April  -  Oeceaber  1987 


Approved  for  public  release:  distribution  is  unlimited. 

liCTEI 


AIR  FORCE  SYSTEMS  COMMAND 
BROOKS  AIR  FORCE  BASE,  TEXAS  78235-$801 


89 


2 


085 


WTICE 


When  Government  drawings,  specifications,  or  other  data  are  used  for  any 
purpose  other  than  In  connection  with  a  definitely  Government-related 
procurement,  the  United  States  Government  Incurs  no  responsibility  or  any 
obligation  whatsoever.  The  fact  that  the  Government  may  have  formulated  or 
in  any  way  supplied  the  said  drawings,  specifications,  or  other  data,  is 
not  to  be  regarded  by  inpl Icatlon,  or  otherwise  In  any  manner  construed,  as 
licensing  the  holder,  or  any  other  person  or  corporation;  or  as  conveying 
any  rights  or  permission  to  nianufacture,  use,  or  sell  any  patented 
Invention  that  may  in  any  way  be  related  thereto. 

The  Public  Affairs  Office  has  reviewed  this  paper,  and  It  Is  releasable  to 
the  National  Technical  Information  Service,  where  It  will  be  available  to 
the  general  public.  Including  foreign  nationals. 

This  paper  has  been  reviewed  and  Is  approved  for  publication. 


HENDRICK  W.  RUCK,  Technical  Advisor 
Training  Systems  Division 


RODGER  D.  BALLEMTINE,  Lt  Col,  USAF 
Chief,  Training  Systems  Division 


U.  REPORT  security  CLASSIFICATION 
Unclassified 


2a.  security  CLASSIFICATION  AUTHORITY 


2b.  DECLASSIFICATION /DOWNGRADING  SCHEDULE 


4.  PERFORMING  ORGANIZATION  REPORT  NUM8ER(S) 
AFHRL-TP -88-39 


REPORT  DOCUMENTATION  PAGE 


Ib  RESTRICTIVE  MARKINGS 


Form  App/ovad 
0MB  No.  0704-0taa 


6a.  NAME  OF  PERFORMING  ORGANIZATION 
Advanced  Technology,  Incorporated 


6c  ADDRESS  (Cty,  Stan,  and  Zl^Coati 
4615  Walzem  Road,  Suite  208 
San  Antonio,  Texas  78218 


8a.  NAME  OF  FUNDING  /  SPONSORING 
ORGANIZATION 

Air  Force  Human  Resources  Laboratory 


3  DISTRIEUTION/AVAILARILITY  OF  REPORT 

Approved  for  public  release;  distribution  is  unlimited. 


S  MONITORING  ORGANIZATION  REPORT  NUM8ER(S) 


7a.  NAME  OF  MONITORING  ORGANIZATION 
Training  Systems  Division 


7b  ADDRESS  (City,  Stan,  and  Z/P  Code! 

Air  Force  Human  Resources  Laboratory 
Brooks  Air  Force  Base,  Texas  78235-5601 


8b.  OFFICE  SYMBOL  9  PROCUREMENT  INSTRUMENT  IDENTIFICATION  NUMBER 
Itf  aooMcabia) 

HO  AFHRL  F41689-86-D-0052 


6b.  OFFICE  SYMBOL 
(If  app/kabie) 


Be  ADDRESS  (City,  Stan,  and  Z/PCodei 
Brooks  Air  Force  Base,  Texas  78235-5601 


10  SOURCE  OF  FUNDING  NUMBERS 


PROGRAM 
ELEMENT  NO 
63227F 


PROJECT 

TASK 

NO. 

NO 

7734 

13 

WORK  UNIT 
ACCESSION  NO 
01 


11.  TITLE  flnc/tnn  Saewnry  Qatttficstion) 

Exploratory  Models  to  Link  Job  Performance  to  Enlistment  Standards 


12.  PERSONAL  AUTHOR(S) 

Gott,  C«0« 


13a.  TYPE  OF  REPORT 
Final 


IB.  SUBJECT  TERMS  (Conrinm  on  revona  if  notttury  and  idontify  tif  biock  nombor) 
aptitude  measurement  Job  performance 

criterion  development  job  performance  measurement 

enlistment  standards  linkage  models  selection/classification  testing 


17.  COSATI  COOES 


FIELD  GROUP  SUB-GROUP 


5  09 


05  08 


19.  ABSTRACT  (CbndntM  on  ravwm  if  notatury  tnri  idanvfy  by  block  nombar) 

j,The  JointrService  Job  Performance  Measurement/Enl istment  Standards  Project  was  initiated  by  Congress  in  1980  to 
develop  job  performance  measures  and  to  subsequently  link  such  measures  to  enlistment  standards  for  military 
jobs.  The  results  of  these  efforts  will  enable  the  services  to  better  answer  vital  questions  of  military  quality 
requirements.  The  progress  to  date  has  primarily  been  in  the  development  of  job  performance  measures.  The  next 
objective  is  to  relate  these  on-the-job  performance  measures  to  enlistment  standards.  This  paper  responds  to  the 
second  objective  by  reviewing  and  evaluating  available  linking  methodologies.  Structured  interviews  were 
conducted  with  personnel  in  key  Air  Force  offices  involved  in  recruiting,  selecting,  and  classifying  enlisted 
personnel  to  discuss  current  procedures  and  candidate  models.  As  a  result,  four  linking  models  were  identified 
and  evaluated:  the  Institutional  expectancy  chart,  break-even  analysis,  the  time  to  proficiency  model,  and  the 
Rand  cost/performance  tradeoff  model.  Each  of  these  models  was  analyzed  within  a  common  evaluative  framework  that 
addressed  required  inputs,  computational  requirements,  quality  of  outputs,  and  strengths  and  weaknesses  of  the 
technique.  Evaluation  results  generally  supported  the  use  of  the  expectancy  chart  and  break-even  analysis  for 
simpler  linkage  goals  (e.g.,  to  determine  the  impact  of  enlistment  standards  on  general  mission  capability), 

■ (Continued) 


20  DISTRIBUTION /availability  OF  ABSTRACT 

(3  UNCLASSIFIED/UNLIMITED  □  SAME  AS  RPT  Q  OTIC  USERS 

21  ABSTRACT  security  CLASSIFICATION 

Unclassified 

22a  NAME  OF  RESPONSIBLE  INDIVIDUAL 

Nancy  J.  All in.  Chief,  STINFO  Branch 

22b.  TELEPHONE  (Include  Area  Code! 
(512)  536-3877 

22c  OFFICE  SYMBOL 

AFHRL/ SC V 

DO  form  1473,  JUN  86 


Fravious  abitionf  ara  obsolete. 


SECURITY  CLASSIFICATION  OF  This  PAGE 


Unclassified 


Item  19  (Concluded): 


whereas  the  time  to  proficiency  and  Rand  models  were  better  suited  for  more  complex  goals 
involving  cost-benefit  ratios  and  more  precise  productivity  indicators.  Reported  conclusions 
emphasized  the  importance  of  (a)  the  goal-setting  process  as  the  first  step  in  pursuing  linkage 
ntethods,  (b)  the  support  from  the  Air  Force  selection  and  classification  community  to  test  and 
implement  any  model,  and  (c)  the  type  of  job  performance  measures  used  (i.e. ,  norm-referenced 
versus  criterion-referenced  tests). 


AFHRL  Technical  Paper  88-39 


Oecenber  1988 


EXPLORATORY  MODELS  TO  LINK  JOB 
PERFORMANCE  TO  ENLISTMENT  STANDARDS 


C.  Deene  Gott 


Advanced  Technology,  Incorporated 
4615  Nalzea  Road 
Suite  208 

San  Antonio,  Texas  78218 
Linder  Subcontract  To: 


Universal  Energy  Systeas,  Incorporated 
Manpower,  Personnel,  and  Training  Division 
4401  Dayton-Xenia  Road 
Dayton,  Ohio  45432-1894 


TRAINING  SYSTEMS  DIVISION 
Brooks  Air  Force  Base,  Texas  78235-5601 


Accession  For 

NT  IS  GRA&I 
DTIC  TAB 
Unannounced 
J’.iStirioatlon. 


By. 


A  .  r  1 1 1 V  Cedes 


Reviewed  and  subaitted  for  publication  by 


Hendrick  W.  Ruck,  Technical  Advisor 
Training  Systeas  Division 


This  publication  Is  primarily  a  working  paper.  It  Is  published  solely  to  document  work  performed. 


SUMMARY 


The  objectives  of  the  Joint-Service  Job  Performance  Measurement/ 
Enlistment  Standards  Project  are  to  develop  on-the-job  performance 
measures  and  to  relate  such  measures  to  the  setting  of  enlistment  stand¬ 
ards  for  job  entry.  The  services’  progress  to  date  has  focused  primarily  on 
the  first  objective.  This  paper  considers  available  methodologies  that  can 
provide  the  linking  mechanism  for  the  second  objective.  Interviews  were 
conducted  with  personnel  in  key  Air  Force  offices  involved  in  recruiting, 
selecting,  and  classifying  enlisted  personnel  in  order  to  better  understand 
the  current  selection  and  classification  process  and  to  discuss  candidate 
methods  and  procedures.  Four  linkage  models  were  identified  and  evalu¬ 
ated:  the  institutional  expectancy  chart,  break-even  analysis,  the  time  to 
proficiency  model,  and  the  Rand  cost/performance  tradeoff  model.  Each  of 
these  models  was  analyzed  with  respect  to  varied  Air  Force  goals  for  link¬ 
ing  performance  to  enlistment  standards  and  in  terms  of  required  input 
data,  efficiency  of  computational  requirements,  and  suitability/quality  of 
output.  The  overall  evaluation  of  the  models  led  to  the  following  general 
conclusions.  The  Air  Force  must  initially  decide  which  goal  to  pursue  from 
among  the  various  possibilities;  e.  g.,  to  validate  the  Armed  Services 
Vocational  Aptitude  Battery  (ASVAB)  scores  as  predictors  of  job  perform¬ 
ance  or,  as  a  more  complex  goal,  to  determine  what  levels  of  ASVAB  pre¬ 
dict  various  levels  of  job  performance  at  what  costs.  Different  models  are 
suitable  for  different  goals.  The  ability  of  available  linkage  models  to 
achieve  Air  Force  goals  depends,  in  general,  on  the  quality  and  type  of  job 
performance  measures  used.  This  paper  is  intended  to  aid  the  decision 
making  process  regarding  suitable  linking  methodologies. 
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PREFACE 


In  response  to  Congress,  the  Department  of  Defense  initiated  job  per¬ 
formance  research  and  development  (R&D)  approximately  5  years  ago  to 
develop  a  technology  for  accurately  measuring  job  performance.  The  Air 
Force  R&D  plan  was  developed  in  coordination  with  the  other  services  to 
ensure  that  all  programs  are  interrelated  and  not  excessively  redundant. 
This  effort  deals  with  one  aspect  of  this  Air  Force  R&D;  namely,  the  meth¬ 
odology  available  to  link  enlistment  standards  to  the  job  performance 
measures  developed. 

The  author  wishes  to  express  sincere  appreciation  to  all  the  individ¬ 
uals  mentioned  in  Appendix  A  who  contributed  to  the  interviewing  phase 
of  this  effort.  In  particular,  Lt  Col  Rodger  Ballentine  of  the  Air  Force 
Human  Resources  Laboratory  provided  very  valuable  guidance  while  ser¬ 
ving  as  the  Laboratory  focal  point  and  extensively  reviewing  and  editing 
early  draft  reports.  He  also  provided  needed  information  and  numerous 
hard-to-find  references  from  his  extensive  personal  library.  The  special 
interest  and  help  given  by  Dr.  Bill  Alley  and  Dr.  Henfc  Ruck  are  appreciated 
as  well.  Finally,  Captain  Eric  Duncan’s  assistance  as  the  project  manager 
was  invaluable  and  the  numerous  insightful  discussions,  critiques,  and 
suggestions  provided  by  Dr.  Sherrie  Gott  are  gratefully  acknowledged. 
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EXPI.ORATORY  MODELS  TO  LINK  JOB  PERFORMANCE 
TO  ENUSTMENT  STANDARDS 


1.  INTRODUCTION 


The  Joint-Service  Job  Performance  Measurement/Enlistment 
Standards  Project  (or  JSP  for  Joint-Service  Project)  has  two  goals:  (a)  to 
investigate  the  feasibility  of  developing  measurement  methodologies  to 
assess  military  job  performance  accurately  and  efficiently  and  (b)  to  link 
enlistment  standards  directly  to  job  performance  once  the  measures  are 
available  (Harris,  1987).  As  it  becomes  increasingly  clear  that  the  first 
objective  is  feasible  and  job  performance  measurement  methods  become 
available,  interest  turns  to  the  second  objective.  This  paper  is  motivated 
by  such  interest. 

The  linkage  process  has  been  described  and  justified  in  the  context  of 
improved  quality  and  utilization  of  the  enlisted  force  as  follows: 

1.  A  definitive  link  between  enlistment  criteria  and  job 
performance  will  enable  us  to  determine  accession  quality 
requirements  with  greater  precision  than  is  now  possible 
(Harris,  1987); 

2.  Hard  evidence  on  the  relationship  between  job  performance 
and  quality  standards  will  allow  us  to  justify  our  recruiting 
resources  on  the  grounds  of  improved  performance  on  the  job 
(Armor,  1987); 

3.  "Linking  enlistment  standards  to  job  performance  can  have 
several  interpretations  ....  Many  of  us  have  now  become 
convinced  of  a  larger  meaning  (larger  than  validation  of 
selection  tests)  for  linking  enlistment  to  job  performance. 

Linking  implies  providing  a  meaning  to  the  scale  of  job  per¬ 
formance  beyond  simple  ranking  of  examinees  ...  to  assist 
policy  makers  in  setting  entrance  standards  ....  Given  the 
strong  competition  for  manpower,  reasonable  and  defensible 
decisions  about  personnel  allocation  depend  on  knowing  the 
resultant  job  performance  that  would  be  experienced.  Differ¬ 
ent  choices  (cutoff  scores)  would  lead  to  better  performance  in 
some  jobs,  not  such  good  performance  in  some  other  jobs  .... 
There  is  no  standard  way  to  make  such  comparisons  ...  no 


accepted  solution.  You  are  at  the  edge  of  the  current  scientific 
methodology."  (Green  &  Shavelson,  1987,  p.  352);  and 

4.  "  I  don't  believe  that  there  is  any  system  that  is  so  good  that  it 

can't  be  improved.  One  of  the  reasons  that  we're  doing  the  Job 
Performance  Measurement  Project  is  so  we  can  improve  the 
way  we  select  and  classify  young  people  into  the  military.  If 
we  sit  back  on  our  laurels  and  say  that  the  system  is  working 
OK,  then  we're  not  ever  going  to  do  it  any  better.  I  am  philo¬ 
sophically  opposed  to  that  notion"  (Sellman,  1987,  p.  419). 

There  are  several  important  and  recurrent  themes  in  these 
characterizations: 


1 .  Linkages  can  improve  decisions  regarding  the  needed  quality 
of  accessions  by  tying  enlistment  criteria  to  the  quality  of  job 
performance: 

2.  Linkages  will  allow  recruiting  resource  budgets  to  be  justified 
in  terms  of  improved  performance  on  the  job; 

3.  Linkages  can  assist  classification  decisions  about  how  to 
allocate  talent  (resources)  across  jobs;  and 

4.  Linkage  implies  giving  meaning  to  job  performance  scales  so 
that  the  consequences  of  lowering  (or  raising)  cutoff  scores  can 
be  explained  to  policy  makers  in  terms  of  impact  on  perform¬ 
ance  on  the  job  and  thus  on  mission  effectiveness. 

The  purpose  of  this  paper  is  to  consider  available  cost/performance 
modeling  techniques  that  can  provide  the  linking  mechanism  for  using  the 
job  performance  measures  to  address  manpower  policy  issues  such  as 
those  stated  above.  The  focus  is  thus  methodological;  more  specifically,  the 
focus  is  on  linking  methodologies.  It  is  assumed  that  the  measurement 
system  developed  and  applied  by  the  Air  Force  as  part  of  the  JSP  is  funda¬ 
mentally  sound,  meeting  traditional  standards  of  reliability  and  validity. 

In  short,  it  is  outside  the  scope  of  this  paper  to  address  the  nature  of  the 
Air  Force  measurement  system  in  depth.  However,  every  attempt  will  be 
made  to  avoid  treating  linking  mechanisms  in  a  vacuum;  rather,  the 
quality  of  the  inputs  needed  from  the  measurement  system  in  order  to 
obtain  useful  and  comprehensible  outputs  from  a  linkage  model  will  be 
carefully  examined.  Similarly,  the  expected  utility  of  linking  methods  for 
Air  Force  manpower  policy  making  will  be  examined  in  the  context  of  the 


current  Air  Force  standard  setting  process.  Thus,  what  the  linking  models 
require  from  job  measurement  methods  will  be  addressed  along  with  how 
the  current  process  might  implement  and  benefit  from  cost/performance 
modeling  techniques. 

To  achieve  these  goals,  a  two-pronged  information  gathering 
approach  was  used  in  this  study.  Literature  in  the  three  major  areas  rele¬ 
vant  to  the  goals  of  the  study  was  reviewed;  namely,  job  entry  standards, 
job  performance  measures,  and  enlistment-performance  linkage  models. 
Secondly,  interviews  were  conducted  with  personnel  in  key  Air  Force 
offices  involved  in  recruiting,  selecting  and  classifying  enlisted  personnel.! 
Results  of  these  efforts  will  be  reported  and  synthesized  in  the  four  major 
sections  that  follow.  First,  the  current  Air  Force  standard  setting/imple¬ 
mentation  will  be  described  and  results  of  the  author's  interviews  reported 
(Section  II).  Next,  the  potential  interplay  of  the  job  performance  measure¬ 
ment  project  and  the  model  building  process  will  be  treated  (Section  III), 
Third,  available  mechanisms  for  achieving  the  interplay  (i.  e.,  linkage 
models)  will  be  reviewed  and  critically  evaluated  (Section  IV).  Finally, 
major  findings  will  be  synthesized  and  recommendations  for  future  work 
will  be  proposed  (Section  V). 

n.  CURRENT  AIR  FORCE  QUALITY  STANDARDS  PROCESS 

How  Standards  Are  Set 


An  understanding  of  how  Air  Force  enlisted  standards  are  currently 
set  (and  adjusted)  is,  of  course,  fundamental  to  the  goals  of  this  effort.  A 
review  of  key  documents  in  the  area  yielded  a  graphic  characterization  of 
the  Air  Force  enlistment/classification  organizational  hierarchy.  The  chart 
shown  in  Figure  1  is  taken  from  Waters,  Laurence,  and  Camara  (1987,  p. 
34)  and  is  intended  to  reflect  how  the  system  handles  problems  that  may 
signal  the  need  for  standard  adjustments.  During  the  author's  interviews 
with  Air  Force  personnel  presently  involved  in  recruiting,  selecting,  and 
classifying  activities,  this  chart  was  used  to  stimulate  discussion  (see  the 


'  Details  regarding  the  special  purpose  interviews  are  provided  in 
Appendixes  A  and  B  and  annotated  references  for  corollary  research  that 
bear  on  the  linkage  of  enlistment  standards  to  measures  of  job 
performance  are  presented  as  Appendix  C. 


Level  4 


Policy  Level  3 
Manage¬ 
ment 


Problem  Level  2 
Analysis 


Problem  Level  1 
Identifica¬ 
tion 


Key;  AFMPC/DPMRTC  HQ  USAF  Classification  and  Training  Branch 

AF/DPXOA  HQ  USAF  Accessions  and  Reenlistment  Policy  Branch 
AFHRL  Air  Force  Human  Resources  Laboratory 

ATC/TT  Air  Training  Command  Technical  Training 

AF/DPPTS  HQ  USAF  Systems/Specialist  Training  Branch 

OASD/FM&P/AP  Directorate  for  Accession  Policy  in  the  Office  of  the 
Assistant  Secretary  of  Defense  (Force  Management 
and  Personnel) 

USMEPCOM  U.  S.  Military  Entrance  Processing  Command 


Note.  From  Waters,  Laurence,  and  Camara  (1987,  p.  34). 

Figure  1.  Air  Force  Selection  and  Classification  Standards 
Management. 
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Questionnaire  in  Appendix  B  for  details).  Participants  in  the  interviews 
provided  solid  verification  —  with  only  minor  modifications  --  regarding 
the  accuracy  of  this  structure. 

The  multistage  process  moves  from  initial  identification  and  analysis 
of  selection/classification-related  problems  (Level  1  and  2)  to  the  culmin¬ 
ating  policy  decisions  (Level  3)  that  are  monitored  by  an  officer  in  the 
Department  of  Defense  (Level  4).  More  specifically,  existing  enlistment 
standards,  which  are  set  forth  in  Air  Force  Regulation  (AFR)  39-1,  are 
reviewed  when  "problems"  in  the  field  are  encountered.  Problems  may 
involve  events  such  as  difficulties  in  meeting  eligibles,  for  example; 
increased  attrition  in  training  courses  as  a  function  of  changes  in  training 
curricula;  adverse  trends  in  reenlistmeni;  and  complaints  from  the  field 
about  unsatisfactory  job  performance.  Other  events  that  might  trigger  a 
review  of  standards  might  include  research  and  development  (R&D)  efforts 
being  carried  out  by  the  Air  Force  Human  Resources  Laboratory  (AFHRL) 
in  areas  such  as  job  learning  difficulty,  or  initiatives  such  as  Rivet 
Workforce  that  would  require  the  restructuring  of  Air  Force  specialties 
(AFSs).  These  influences  on  the  process  may  be  surfaced  by-  recruiters, 
trainers,  personnel  in  the  major  commands,  researchers.  Air  Staff,  or 
others.  After  the  identification  stage,  such  problems  are  channeled  to  an 
Air  Force  functional  manager,  who  is  responsible  for  the  specialty  in  which 
the  problem  exists  (Level  2/Problem  Analysis).  The  functional  managers 
collect  information,  analyze  the  situation,  and  decide  what  action(s)  to  take. 

In  making  these  determinations,  the  functional  managers  should 
keep  in  mind  that  aptitude  requirements  serve  many  purposes.  In 
addition  to  the  major  functions  of  screening  candidates  for  enlistment  and 
matching  persons  to  jobs,  other  functions  of  aptitude  requirements  — 
identified  by  Maginnis,  Uchima,  and  Smith  (1975a)  as  being  equally 
important  -  include  such  areas  as  career  development,  assignment  flexi¬ 
bility,  and  job/service  satisfaction.  If  aptitude  requirements  are  set  too 
low,  abundant  manpower  for  the  Air  Force  may  be  available,  but  training 
problems  and  poor  job  performance  may  result.  If  aptitude  requirements 
are  set  too  high,  the  available  manpower  pool  might  be  limited,  and  under¬ 
utilization  and  job  dissatisfaction  may  result.  In  either  case,  career  devel¬ 
opment,  retention,  and  classification  are  affected.  In  their  three-volume 
report,  Maginnis  et  al.  (1975a,  1975b,  1975c)  traced  the  history  of  Air 
Force  aptitude  levels  and  their  impacts  on  the  personnel  system.  The 
authors  pointed  out  that  "expert  judgment"  apparently  played  a  large  part 
in  the  original  setting  of  aptitude  minimums  for  entry  into  specific  AFSs. 
They  then  described  the  impacts  of  aptitude  requirements  in  several 
hypothetical  situations;  i.  e.,  "Set  I"  where  aptitude  levels  are  set  higher- 
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than-optimal  for  entry-level  job  performance  effectiveness,  "Set  2"  where 
aptitude  scores  are  set  lower- than-optimal,  and  "Set  3"  where  aptitude 
levels  are  set  equal-to-optimal.  Impacts  are  expressed  in  terms  of  the 
manpower  pool;  recruitment,  selection,  and  enlistment;  classification  and 
assignments;  training;  job  performance;  job  satisfaction  and  morale;  promo¬ 
tion  and  transfer;  and  retention.  Comparisons  of  impacts  revealed  that 
selection  using  lower-than-optimal  requirements  shows  more  negative 
impacts  than  does  selection  using  higher-than-optimal  requirements. 
Results  of  the  Maginnis  et  al.  (1975a,  1975b,  1975c)  research  are  men¬ 
tioned  here  to  illustrate  the  availability  of  information  and  procedures  (at 
least  hypothetically)  to  assist  functional  managers  and  offices  in  the 
current  standards  setting  process. 

The  Classification  and  Training  Branch  at  the  Air  Force  Military 
Personnel  Center  (AFMPC/DPMRPQ)  is  the  focal  point  and  key  office  for 
coordinating  AFR  39-1  (enlisted)  and  AFR  36-1  (officer)  changes.  An 
extensive  coordination  package  is  developed  and  the  coordination  process 
begins.  Offices  that  are  involved  in  the  coordination  loop  include  the 
following: 


HQ  AFMEA/MEXR 

(Air  Force  Management  Engineering 
Agency-provides  coordination  on 
manpower  implications) 

ANGSC/MPP 

(Air  National  Guard  Service  Center- 
ensures  appropriate  Air  National  Guard 
coordination) 

HQ  USAF/REP 

(Air  Force  Reserve  Personnel-ensures 
appropriate  Air  Force  Reserve 
coordination) 

HQ  AFMPC/DPMR 

(Air  Force  Military  Personnel  Center- 
provides  airmen  and  officer  assign¬ 
ments  coordination) 

HQ  USAF/DPPP 

(Air  Staff-provides  coordination  on 
force  programs  [Trained  Personnel 
Requirement]) 

HQ  USAF/DPPTF 

(Air  Staff-provides  coordination  for 
flying  specialties) 

HQ  USAF/DPPTS 

(Air  Staff-provides  coordination  for 
nonrated  specialties) 

HQ  USAF/DPXOA 

(Air  Staff-provides  accessions  and 
airmen  reenlistment  policy 
coordination) 

HQ  ATC/TTSA 


HQ  AFMPC/DPMAPA 

HQ  USAF/DPPH 

HQ  AFMPC/DPMRPP 

HQ  USAF/DPXF 
HQ  USAF/DPXOP 
HQ  USAF/DPXE 
HQ  AFMPC/SGM 

AFOSP/SPP 

HQ  AFMPC/DPMRAS 

HQ  AFMPC/DPMRC 

HQ  AFAAMRL/HEB 


(Air  Force  Air  Training  Command- 
provides  training  and  recruiting 
coordination) 

(Air  Force  Military  Personnel  Center- 
provides  coordination  for  programs 
management) 

(Air  Staff-provides  coordination  for 
human  relations,  strength  and  aptitude 
requirements,  and  woman  issues) 

(Air  Force  Military  Personnel  Center- 
provides  assignment  policy  coordin¬ 
ation) 

(Air  Staff-provides  coordination  for 
force  structure  plans) 

(Air  Staff-provides  promotions  and 
career  force  coordination) 

(Air  Staff-provides  entitlements 
coordination) 

(Air  Force  Military  Personnel  Center- 
provides  medical  and  physical 
standards  coordination) 

(Air  Force  Office  of  Security  Police- 
provides  security  and  security  clearance 
coordination) 

(Air  Force  Military  Personnel  Center- 
provides  skills  management  and 
retraining  coordination) 

(Air  Force  Military  Personnel  Center- 
provides  coordination  on  issues 
pertaining  to  E-9s  [Chief  Master 
Sergeants]) 

(Air  Force  Armstrong  Aeromedical 
Research  Laboratory-provides 
coordination  for  strength  and  aptitude 
requirements  and  new  AFSCs) 


Action  taken  by  the  various  offices  within  the  coordination  chain 
depends  on  the  nature  and  priority  of  the  problem  involved.  Conceivably, 
a  Request  for  Personnel  Research  to  AFHRL  might  be  initiated,  if  the  pro¬ 
blem  warranted  research  attention.  One  methodology  available  to  the 
various  offices  within  the  elaborate  coordination  process  is  based  on  the 
theory  of  occupational  learning  difficulty.  This  methodology,  described  by 
Weeks  (1984),  can  be  used  to  evaluate  aptitude  requirement  minimums 


for  Air  Force  enlisted  specialties.  A  jasic  assumption  of  the  method  is  that 
aptitude  (i.  e.,  aptitude  minimums)  and  learning  time  (i.  e.,  task  learning 
difficulty)  are  related.  A  standardized  measurement  procedure,  described 
by  Burtch,  Lipscomb,  and  Wissman  (1982),  uses  occupational  survey  data 
collected  by  the  the  Air  Force  Occupational  Measurement  Center  to  derive 
indices  of  occupational  learning  difficulty  for  each  specialty.  Such  mea¬ 
sures  provide  an  empirical,  job-centered  frame  of  reference  which  can  be 
used  to  determine  the  relative  order  of  aptitude  requirement  minimums 
and  support  other  critical  management  decisions.  Results  of  occupational 
learning  difficulty  R&D  by  AFHRL  have  led  to  aptitude  requirement  adjust¬ 
ments  which  were  accomplished  and  published  in  the  1982  revision  of  AFR 
39-1. 

Examination  of  the  above  list  of  agencies  reveals  the  scope  and 
impact  of  standard  setting  in  the  sense  that  most  aspects  of  the  life  cycle  of 
the  enlisted  forces  are  included  in  the  coordination  process;  i.  e.,  recruit¬ 
ing,  classifying,  training,  assigning,  promoting,  utilizing,  etc.  Once  this 
lengthy  coordination  phase  is  complete,  the  results  are  forwarded  to  the 
"Policy  Management/Level  3"  (HQ  USAF/DPXOA)  for  final  approval  before 
eventual  implementation.  At  this  point  in  the  process,  the  proposed  apti¬ 
tude  minimum  is  customarily  reviewed  again  for  consistency  and  reason¬ 
ableness  in  terms  of  AFR  39-1.  Also,  any  additional  statistical  data  which 
might  aid  in  the  more  accurate  setting  of  standards,  such  as  current  estim¬ 
ated  distributions  of  the  available  manpower  pool,  would  be  considered  at 
this  time. 

Given  the  scope  and  implications  of  the  process  described  above,  one 
could  imagine  that  it  would  be  useful  to  the  offices  involved  to  have  a 
means  by  which  the  consequences  of  various  policy  decisions  could  be 
projected  and  evaluated.  For  example,  if  standards  were  raised  by  10 
points  for  an  entire  career  field,  or  even  for  a  given  AFS,  what  impact 
would  the  decision  have  on  the  recruiting  quotas  for  the  rest  of  the  force? 
What  would  be  the  impact  on  recruiting  cost,  if  the  goal  were  to  increase 
the  number  of  high  ability  personnel?  And  what  about  the  impact  on 
training?  Decision  modeling  techniques  could  conceivably  serve  a  very 
useful  purpose  during  standard  setting  by  allowing  the  various  conse¬ 
quences  of  such  decisions  to  be  determined  and  evaluated  for  acceptability 
before  implementation.  The  case  for  modeling  cost  and  personnel  out¬ 
comes  becomes  even  stronger  during  times  of  limited  resources  when  both 
personnel  and  financial  resources  are  at  risk.  The  case  becomes  stronger 
yet  if  one  examines  the  impact  of  less-than-optimal  standards  on  later 
stages  in  the  classification-training  pipeline.  The  impact  on  classification  is 
examined  next. 
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Implementation  of  Enlistment  Standards 
in  the  Matching  of  Personnel  to  Jobs 


Each  year  approximately  56,000  enlisted  personnel  are  selected  and 
classified  into  roughly  300  occupations  in  the  Air  Force.  Two  computer- 
based  systems  accomplish  this  staggering  task.  The  first,  the  Procurement 
Management  Information  System  (PROMIS),  is  a  pre-enlistment  system 
which  processes  nonprior-service  recruits  from  the  time  they  become 
interested  in  joining  the  Air  Force  until  the  time  they  enter  basic  military 
training  (BMT).  The  second,  the  Processing  and  Classification  of  Enlistees 
(PACE)  system  is  a  post-enlistment  system  which  classifies  enlistees  during 
BMT. 


The  basic  concept  of  PROMIS  and  PACE  is  to  select  and  classify  air¬ 
men  by  optimally  assigning  them  to  jobs,  given  their  particular  aptitudes 
at  the  time  of  entry  and  the  goals  and  constraints  of  the  Air  Force  express¬ 
ed  in  terms  of  the  training  slots,  manpower  authorizations,  etc.  A  decision 
modeling  technique  called  policy  specifying  (Ward,  1977)  was  developed 
at  AFHRL  to  achieve  the  optimal  person-job  match  (PJM).  The  technique 
takes  variables  identified  as  pertinent  to  the  Air  Force  systems  --  includ¬ 
ing  aptitude  indicators  and  training/job  openings  --  and  combines  them 
into  a  single  "payoff  value"  for  a  given  recruit.  A  payoff  value  indicates 
the  worth  to  the  Air  Force  of  classifying  a  recruit  into  a  particular  job.  The 
PJM  process  is  summarized  in  Figure  2. 

Both  PROMIS  and  PACE  take  as  givens  the  aptitude  scores  from 
"Personnel  Files"  (as  shown  in  Figure  2)  and  the  job  entry  requirements 
from  "Job  Files"  and  then  process  them  to  maximize  the  payoff  values  of 
incoming  personnel  through  optimal  assignments.  The  "payoff  algorithm" 
shown  in  Figure  2  uses  the  policy  specifying  methodology  to  compute  pay¬ 
off  values  that  reflect  current  Air  Force  judgments  concerning  individual 
job  assignments.  A  separate  equation  for  each  job  is  used  by  the  algorithm 
to  compute  a  payoff  value  for  each  individual  recruit  for  all  available  jobs. 
These  payoff  values  are  the  key  inputs  for  the  actual  assignment  of  per¬ 
sonnel  to  jobs,  which  takes  place  next  in  the  "assignment  algorithm."  For  a 
more  thorough  discussion  of  the  PJM  algorithm  refer  to  Ward  (1977).  It  is 
in  the  computations  of  the  payoff  values  that  the  minimum  standards 
described  in  the  previous  section  play  an  important  role.  Both  enlistment 
standards  and  individual  aptitude  indicators  are  used  as  key  decision 
variables  in  the  mathematical  models  and  equations  used  to  derive  payoff 
values.  The  use  of  these  payoff  values  enables  the  Air  Force  to  better 
match  personnel  graduating  from  BMT  with  AFSs  based  on  a  policy  defined 
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jgure  2.  TIie  Person-Job  Match  Process. 
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by  Air  Force  policy  makers.  In  addition,  the  PJM  process  plays  an  impor¬ 
tant  role  in  the  distribution  of  aptitude  levels  (quality)  in  a  particular 
specialty.  An  algorithm  that  de^s  with  the  standard  setting  process  for 
AFSs  should  take  this  actual  resultant  distribution  into  account  when 
considering  the  supply  of  available  candidates. 

The  Need  for  Linkage  Models 
to  Improve  the  Quality  of  Air  Force  Accessions 


Given  the  selection  and  classification  procedure  previously  described 
and  the  fact  that  the  Air  Force  has  within  recent  years  been  satisfactorily 
meeting  all  of  its  needs  in  terms  of  both  quantity  and  quality  of  accessions 
(recruits),  some  have  questioned  the  basic  need  for  linkage  models  to  im¬ 
prove  the  selection  process.  In  this  vein,  there  are  those  who  would  argue 
that  because  the  Air  Force  is  presently  in  an  optimal  recruiting  position, 
the  system  either  perfectly  selects  to  ensure  competence  or  screens  out 
any  false  positives  that  might  be  erroneously  selected.  To  those  that  hold 
this  view,  the  Joint-Service  Job  Performance  Measurement/  Enlistment 
Standards  Project  is  probably  seen,  first  and  foremost,  as  a  means  by 
which  this  optimal  state  of  affairs  (i.  e.,  highest  quality  recruits  and  no 
incompetents  on  the  job)  is  verified.  To  them,  the  JSP  purpose  is  to  "prove 
that  what  we're  doing  now  is  OK  ...  the  answer  at  the  end  (of  the  re¬ 
search)  is  (to  be)  a  check  mark,  (that)  everything's  fine"  (Strickland,  1987, 
p.  413).  Their  interpretation  of  linking  enlistment  standards  to  job  per¬ 
formance  would  be  a  linkage  model  that  simply  computes  a  validity 
coefficient  between  predictor  (the  Armed  Services  Vocational  Aptitude 
Battery  [ASVAB])  and  criterion  (job  performance)  measures. 

Other  levels  of  interpretation  are  also  possible,  however.  They  range 
from  the  position  supported  by  Harris  (1987),  which  is  to  determine 
accession  quality  requirements  with  greater  precision  than  is  presently 
possible;  to  that  implied  by  Armor  (1987)  and  Sellman  (1987),  which  is  to 
justify  the  need  and  associated  costs  for  high  quality  accessions  in  terms  of 
impact  on  job  performance;  to  that  embraced  by  Green  and  Shavelson 
(1987)  and  others  on  the  National  Academy  of  Sciences  Technical  Oversight 
Committee  to  the  JSP.  In  the  latter  view,  a  linkage  model  should  be  used 
to  improve  selection  and  classification  decisions  and  to  generally  give 
meaning  to  job  performance  scales  by  explaining  both  the  requirements 
that  account  for  competent  job  performance  and  the  chain  of  relationships 
between  the  aptitudes  tapped  by  selection  tests  and  the  performance 
demands  encountered  by  personnel  (after  training)  on  the  job. 


In  the  next  two  sections  of  this  paper,  linkage  approaches  will  be 
described  that  would  seem  appropriate,  given  these  various  interpreta¬ 
tions,  or  goals.  First,  the  common  elements  in  candidate  linkage  models 
and  the  model  building  process  in  general  will  be  discussed. 

m.  THE  LINKAGE  MODEL  BUILDING  PROCESS 
WITHIN  THE  JOINT-SERVICE  PROJECT  (JSP)  FRAMEWORK 


The  four  linkage  methodologies  to  be  presented  in  Section  IV  share 
certain  common  elements  and  fundamental  processes,  such  as  optimization 
formulations.  The  common  elements  include  the  output  or  the  specific 
objective(s)  to  be  accomplished,  the  criteria  (e.g.,  constraints)  to  be  invoked 
in  meeting  those  objectives,  and  the  input  or  data  elements  to  be  linked 
(e.g.,  aptitude  and  job  performance  measures).  These  elements  will  be 
described  in  this  section  in  the  context  of  a  model  building  scheme. 


Identification  of  an  Appropriate  Objective 
and  Optimization  Criteria 

For  most  of  the  linkage  models  considered  in  this  paper,  the  nature 
of  the  output  is  the  same;  namely,  an  enlistment  standard  (or  set  of  stand¬ 
ards)  as  a  cutoff  score  for  selection  decisions.  The  manner  in  which  the 
cutoff  score  is  derived  computationally  is  a  function  of  the  parameters 
included  in  the  linkage  model.  In  turn,  the  objective  or  purpose  to  be  met 
by  the  linkage  model  fundamentally  drives  the  selection  of  model  para¬ 
meters.  Possible  objectives  for  the  Air  Force  in  this  regard  might  include 
the  following: 

1 .  To  determine  the  validity  of  AS  VAB  aptitude  measures  as 
predictors  of  actual  performance  on  the  job; 

2.  To  determine  what  levels  of  AS  VAB  performance  will  predict 
various  levels  of  job  performance: 

3.  To  determine  what  levels  of  AS  VAB  performance  will  predict 
various  levels  of  job  performance  at  specified  costs:  and 

4.  To  derive  the  optimal  allocation  of  personnel  across  Air  Force 
specialties  in  terms  of  cost  and  utilization  of  talent. 
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Optimization  criteria  related  to  these  objectives  generally  concern 
factors  such  as  cost,  performance  or  productivity,  mental  quality 
(aptitude),  time,  and  force  size.  For  example,  candidate  optimization  goals 
might  be: 

1.  To  maximize  the  quality  of  performance; 

2.  To  minimize  costs  to  achieve  an  acceptable  level  of 
performance  quality; 

3.  To  minimize  time  (in  training  and  experience)  to  achieve  an 
acceptable  level  of  performance  quality; 

4.  To  minimize  force  size:  or 

5.  To  maximize  return  on  (personnel)  investment  (ROI). 

Maximizing  performance  quality  can  be  interpreted  as  building  the 
most  capable  force.  In  order  to  produce  realistic  results,  this  criterion 
must  be  constrained  to  provide  a  ceiling  on  force  cost.  In  addition,  the 
user  may  wish  to  set  other  constraints  such  as  force  size  and  experience 
levels. 

A  minimum  cost  procedure,  in  contrast,  will  require  constraints  on 
both  performance  and  force  size  in  order  to  assure  some  acceptable  level 
of  capability.  As  before,  experience  levels  can  also  be  constrained.  Mini¬ 
mizing  force  size  will  generally  produce  the  same  results  and  similarly 
requires  constraints  to  provide  lower  limits  on  both  capability  and 
experience. 

The  final  criterion,  ROI,  is  a  hybrid  of  cost  and  performance.  It  can 
be  formulated  either  to  maximize  benefit  per  unit  of  cost  such  as  produc¬ 
tive  unit/dollar  or,  the  inverse,  to  minimize  cost  per  unit  of  benefit  (i.  e., 
cost/productive  unit).  The  results  are  equivalent  in  either  case.  The 
advantage  of  using  a  cost/  benefit  criterion  is  that  the  issues  of  perform¬ 
ance  (effectiveness)  and  cost  (efficiency)  are  addressed  directly  in  the 
objective  function  rather  than  through  constraints,  thereby  assuring  a 
broader  range  of  feasible  solutions.  The  difficulty  in  formulating  con¬ 
straints  in  these  variables  lies  in  determining  adequate  or  acceptable 
levels.  The  ROI  formulation  is  generally  preferred  in  this  context  since 
decision  makers  are  presented  with  the  most  efficient  "product”  per  unit 


benefit  (i.  e.,  the  most  "cost  effective"  solution).  Additional  capability  may 
then  be  "purchased"  at  an  easily  identifiable  marginal  cost  if  it  is  required. 

From  the  Air  Force's  perspective,  the  preferred  optimization  would 
likely  mean  establishing  the  highest  quality  force  that  is  reasonably 
attainable.  From  Congress’  perspective,  optimization  would  likely  mean 
minimizing  costs  to  man  a  mission-capable  force.  For  the  Air  Force  to 
defend  its  need  for  specific  quality  standards  at  certain  levels  of  cost,  it  is 
necessary,  according  to  some  viewpoints,  to  articulate  how  mission  capa¬ 
bility  and  thus  readiness  are  affected  by  lowered  standards  (Armor,  1987; 
Sellman,  1987).  This  requirement  in  turn  dictates  certain  types  of  input 
variables  as  performance  measures. 


The  Input  Data  Elements  and  Empirical  Linkages 


Inputs  to  standard  linkage  models  can  be  loosely  grouped  into  two 
general  categories  —  cost  factors  and  individual  capability  factors.  In  the 
first  category,  factors  such  as  the  costs  of  recruiting,  training,  and  main¬ 
taining  the  enlisted  force  are  relevant,  as  well  as  costs  expressed  as  "lime 
to  proficiency"  or  rate  of  experience  gains.  Typical  objective  functions 
involve  minimizing  costs  such  as  these,  while  maximizing  the  capability 
factors  in  the  second  category  (e.  g.,  productivity  and  aptitude).  Measures 
of  aptitude  are  available  as  ASVAB  composites.  Measures  of  job  perform¬ 
ance,  expressed  as  level  of  productivity  or  proficiency,  must  be  obtained. 
Also,  the  empirical  linkages  between  the  aptitude  and  performance  mea¬ 
sures  are  important  to  the  meaningfuiness  and  comprehensibility  of  the 
outputs  obtained  from  the  various  linkage  models.  The  issues  related  to 
job  performance  measures  as  input  and  their  empirical  linkages  to  enlist¬ 
ment  standards  are  considered  next. 

Types  of  Job  Performance  Measures 

Given  the  variability  in  the  objectives  or  purposes  to  be  served  by 
linking  enlistment  standards  to  measures  of  job  performance,  different 
types  of  job  performance  measures  are  applicable.  In  the  previous  section 
where  objectives  were  discussed,  two  general  goals  were  established: 


1.  To  validate  the  ASVAB  against  measures  of  on-the-job 
performance;  and 


2. 


To  determine  the  relationship  between  different  levels  of 
ASVAB  scores  and  different  levels  of  job  performance  so  that 
enlistment  standards  can  be: 


made  more  precise, 

b.  justified  in  terms  of  impact  on  job  performance  (and  thus 
mission),  and 

c.  optimized  so  as  to  minimize  costs  while  fielding  a  capable 
enlisted  force. 

If  goal  1  is  selected  bv  the  Air  Force,  then  the  inputs  required  as 
measures  of  job  performance  can  be  either; 

1.  Norm-referenced  test  scores  expressed  as  percentiles  (as  are 
ASVAB  composites)  indicating  an  airman's  relative  ranking 
with  respect  to  others  tested,  or 

2.  Criterion-referenced  test  scores  expressed  as  the  airman’s 
degree  of  mastery  of  the  job  (i.  e.,  the  "criterion  being 
referenced"). 

For  example,  goal  1  could  presumably  be  achieved  by  simply  correl¬ 
ating  norm-referenced  ASVAB  scores  with  norm-referenced  job  perform¬ 
ance  measures,  such  as  scores  on  the  Walk-Through  Performance  Testing 
(WTPT)  component  (Hedge  &  Teachout,  1986).  If  the  validity  coefficient  is 
acceptable  according  to  some  prespecified  standard  (e.  g.,  significantly 
greater  than  zero),  then  the  validation  purpose  of  goal  1  is  achieved. 

However,  if  the  coefficient  or  set  of  coefficients  is  viewed  as  low, 
which  is  a  possible  interpretation  of  results  reported  in  a  recent  study  by 
Dickinson,  Hedge,  and  Ballentine  (1987).  then  the  results  may  require 
further  explanation.  Given  the  well-entrenched  position  of  the  ASVAB  in 
military  selection  procedures,  the  probable  interpretation  of  such  low 
validity  coefficients  would  be  to  indict  the  job  performance  tests.  On  the 
other  hand,  the  relationship  between  ASVAB  predictors  and  job  perform¬ 
ance  criteria  could  be  further  investigated.  In  addition,  a  more  precise 
characterization  of  the  relationship  between  ASVAB  components  and  per¬ 
formance  test  results  for  specific  areas  of  job  content  would  assist  in  the 
future  development  of  valid  aptitude  tests.  The  following  example  is 
offered  to  make  the  point  more  concrete. 


Dickinson  et  al.  (1987)  reported  eight  specific  areas  of  technical 
proficiency  for  the  jet  engine  mechanic  specialty,  which  were  derived 
through  analyses  of  occupational  survey  data  and  judgments  from  subject- 
matter  experts: 

1.  Complete  forms; 

2.  Prepare  engine  for  storage/shipping; 

3.  Inspect  engine; 

4.  Remove/replace  engine  components; 

5.  Conduct  quality  control  inspections; 

6.  Perform  engine  maintenance  in  shop  environment; 

7.  Perform  engine  maintenance  on  flightline;  and 

8.  Troubleshoot  engine  malfunctions. 

If  the  present  performance  data  set  on  the  jet  engine  mechanics 
permitted,  job  area  subtest  scores  could  be  computed  for  airmen  on  these 
individual  job  content  areas  or  on  logical  composites  of  the  areas  listed 
above  such  as  1-2,  3-5,  or  6-8.  This  approach  is  one  of  several  recom¬ 
mended  by  Wigdor  and  Green  (1986,  p.  57)  to  better  understand  ”.  .  .  the 

chain  of  relationships  between  the  abilities  tapped  by  selection  tests  and 
performance,  after  training,  on  the  job." 

Analyses  to  relate  ASVAB  predictors  to  job  performance  criteria 
expressed  as  job  content  area  subtests  could  reasonably  be  expected  to 
provide  valuable  information  for  aptitude  test  development.  For  example, 
given  the  patterns  of  correlations  reported  by  Dickinson  et  al.  (1987)  and 
the  known  academic  nature  of  ASVAB  and  technical  school  measures  (i.  e., 
academic  in  the  sense  that  general  verbal  and  quantitative  components  are 
heavily  weighted),  it  seems  reasonable  to  expect  stronger  relationships 
between  ASVAB  scores  and  our  jet  engine  subtests  denoted  1  and  2,  and 
possibly  3  and  4,  than  with  other  job  areas.  In  job  areas  1-4,  one  would 
expect  performance  demands  to  be  largely  the  following  of  routine,  well- 
established  procedures  that  would  logically  be  predicted  by  general  verbal 
and  quantitative  competencies  (i.  e.,  ASVAB  scores).  On  the  other  hand, 
general  academic  competencies  may  not  predict  complex  problem  solving 
skills  in  specialized  subject-matter  domains,  such  as  required  by  the 


troubleshooting  and  shop/flightline  maintenance  content  areas  (subtests  6- 
8).  Tasks  in  those  subdomains  are  often  ill-structured,  meaning  no  well- 
specified  procedures  for  solution  are  available  (Gott,  Lesgold,  &  Glaser, 

1987) . 

With  this  finer-grained  validity  approach,  a  reasonable  outcome 
might  be  sufficiently  high  correlations  between  ASVAB  scores  and  per¬ 
formance  in  certain  job  areas  (e.g.,  1-4),  and  low  correlations  in  others. 
Areas  where  the  relationships  are  low  would  suggest  that  there  are  per¬ 
formance  demands  (or  behavioral  requirements,  to  use  a  term  from  the 
National  Academy  of  Sciences  Committee)  associated  with  the  job  that  do 
not  map  well  onto  traditional  aptitude  indices.  In  point  of  fact,  it  is  this 
very  hypothesis  that  has  motivated  AFHRL’s  Learning  Abilities  Measure¬ 
ment  Program  (LAMP).  The  goal  of  that  program  (Kyllonen  &  Christai, 

1988)  is  to  develop  a  theory  of  learning  ability  that  would  lead  to  new 
selection  measures  to  supplement  the  ASVAB  in  order  to  improve  its 
predictive  validity,  especially  in  Air  Force  specialties  where  information 
processing  demands  are  heavy.  . 

ILgoal  2  is  selected  bv  the  Air  Force,  then  the  inputs  required  as 
measures  of  job  performance  should  be  criterion-referenced  test  scores. 
Mastery  scores  by  definition  use  a  scale  that  gives  meaning  to  the  per¬ 
formance  data  because  an  individual’s  degree  of  mastery  of  the  specialty 
in  question  is  estimated.  Without  such  a  scaling  of  performance  scores,  it 
would  be  impossible  to  model  and  estimate  the  impact  that  a  change  in 
enlistment  standards  would  have  on  job  performance  capability. 

To  summarize,  while  norm-referenced  job  performance  tests  will 
satisfy  the  basic  linkage  goal  (goal  1)  of  validating  the  ASVAB,  the  inter- 
pretability  of  low  correlations,  should  they  occur,  would  be  problematic. 
Further  clarification  of  the  relationship  between  ASVAB  and  performance 
criteria  could  be  gained  by  generating  performance  subtest  scores  in 
specific  job  content  areas  and  using  such  performance  scores  to  validate 
present  or  proposed  aptitude  tests.  This  approach  would  provide  invalu¬ 
able  information  for  those  attempting  to  develop  future  aptitude  tests  that 
predict  on-the-job  performance.  Certain  job  content  areas  may  be  predict¬ 
ed  quite  well  by  ASVAB  scores,  suggesting  that  the  right  aptitude  con¬ 
structs  are  being  tapped  in  the  recruiting  population.  Conversely,  other 
areas  may  be  poorly  predicted,  suggesting  the  need  for  research  such  as 
the  LAMP  program  to  improve  selection  measures.  On  the  other  hand,  to 
determine  the  relationship  between  different  levels  of  ASVAB  scores  and 
Job  performance  (i.  e.,  setting  enlistment  standards),  criterion-referenced 
scoring  is  recommended. 
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The  foregoing  account  argues  for  aptitude-performance  linkages  that 
can  be  supported  either  theoretically  or  logically.  Finding  relationships 
between  aptitude  tests  and  performance  criteria  can  serve  to  strengthen 
the  validity  of  ASVAB  as  a  predictor  of  job  performance  and  suggests  the 
need  for  more  detailed  investigation  of  the  aptitude-performance  relation¬ 
ship.  Detailed  investigations  of  the  relationship  between  job  content  areas 
such  as  completing  forms  or  troubleshooting  faulty  engines  and  the  under¬ 
lying  skills  and  abilities  sought  in  the  recruit  population  serve  to  identify 
areas  for  future  aptitude-job  performance  R(&D.  Establishing  the  relation¬ 
ship  between  job  mastery  and  recruit  aptitudes  allows  enlistment  stand¬ 
ards  to  be  justified  in  terms  of  how  airman  performance  would  be  adver¬ 
sely  impacted  by  lowered  standards  (Armor,  1987;  Sellman,  1987).  Per¬ 
haps  more  importantly,  military  quality  requirements  can  be  better 
understood  if  job  proficiency  can  be  precisely  tied  to  selection  tests.  This 
goal  requires  a  combination  of  efforts  to  maximize  the  way  existing  tests 
(ASVAB)  predict  job  mastery  and  to  develop  aptitude  tests  that  better 
predict  job  proficiency. 


Criteria  for  Evaluating  Linkage  Models 


The  previous  discussion  presented  the  various  goals  the  Air  Force 
might  pursue  through  linking  enlistment  standards  to  performance  mea¬ 
sures,  as  well  as  the  elements  and  processes  shared  by  available  linkage 
models.  Given  that  discussion,  it  is  now  possible  to  propose  a  set  of  criteria 
to  use  as  a  basis  for  model  evaluation. 

First,  the  Air  Force  must  decide  which  goal  is  to  be  served  in  the 
linking  of  selection  standards  to  job  performance.  Model  parameters  are 
dictated  by  whether  the  goal  is  to  compute  a  simple  validity  coefficient  or 
to  do  more  elaborate  analyses  requiring  cost  and  benefit  factors.  The 
acceptability  of  the  model  to  the  Air  Force,  as  well  as  its  ultimate  imple¬ 
mentation,  are  directly  related  to  how  well  the  model  serves  its  intended 
purpose(s).  Accordingly,  model  evaluation  is  meaningful  only  in  terms  of 
how  well  it  meets  the  stated  purpose(s). 

Once  the  goal  is  established,  it  becomes  possible  to  evaluate  each 
model  with  respect  to  the  specified  goal  as  well  as  against  additional 
general  criteria  as  follows; 


1.  What  is  the  quality  of  the  required  inputs  and  are  they  easily 


Qhiainabla? 

2.  Are  the  computational  requirements  associated  with  the  model 
(including  optimization  processes)  efficient  and  flexible? 

3.  Does  the  output  serve  the  stated  purpose?  Is  it  easily 
interpretable? 

To  illustrate  how  these  general  criteria  would  apply  in  specific  cases, 
several  examples  are  presented. 

1 .  Example  No.  1:  GOAL  is  ASVAB  iob  performance  validation 
only.  This  example  falls  at  the  low  end  of  the  complexity 
continuum  with  respect  to  model  parameters  and  computa¬ 
tional  requirements.  As  proposed  earlier  in  this  section,  norm- 
referenced  ASVAB  scores  would  provide  suitable  input  as 
predictor  variables,  and  norm-referenced  job  performance 
measures  would  be  suitable  as  criterion  scores.  Computational 
requirements  are  minimal  since  what  a  simple  validation 
model  entails  is  the  calculation  of  a  validity  coefficient  as  out¬ 
put.  The  validity  coefficient  would  establish  the  predictive 
validity  of  the  ASVAB  and  thereby  meet  the  model’s  stated 
purpose.  The  output  would  not  reflect  the  quality  of  job  per¬ 
formance  associated  with  various  levels  of  aptitude;  thus,  the 
interpretation  of  the  predictor-criterion  relationship  would  be 
very  circumscribed. 

2.  Example  No.  2:  GOAL  is  to  determine  enlistment  standards  for 
a  given  job  that  minimize  the  cost  of  achieving  given  levels  of 
iob  performance.  This  example  falls  at  the  high  end  of  the 
complexity  continuum  with  respect  to  model  parameters  and 
computational  requirements.  Required  inputs  would  include 
the  levels  of  job  performance  to  be  attained  (e.g.,  minimally 
acceptable  score  on  a  job  performance  test);  the  associated 
aptitude  scores;  and  various  cost  estimates  covering  recruiting, 
training,  and  force  maintenance/retention.  Many  of  these 
input  variables  might  be  difficult  to  obtain;  e.  g.,  the  minimal 
level  of  acceptable  job  performance  (especially  if  only  norm- 
referenced  test  data  were  available)  and  the  costs  of  recruiting. 

The  evaluative  framework  described  previously  will  be  applied  to 
the  specific  models  reviewed  in  the  following  section. 
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IV.  REVIEW  OF  LINKAGE  METHODOLOGIES 


Four  major  techniques  have  been  identified  as  having  potential 
applicability  to  the  Air  Force’s  linkage  objective(s):  expectancy  charts, 
break-even  analysis,  a  time  to  proficiency  model,  and  the  Rand  cost/ 
performance  tradeoff  model. 


Expectancy  Charts 


Description 

The  institutional  expectancy  chart  method  is  one  of  two  possible 
linkage  techniques  described  by  Cascio  (1987)  in  a  paper  prepared  for  the 
Committee  on  the  Performance  of  Military  Personnel.  The  basic  objective 
of  the  method  is  to  depict  the  likelihood  of  "successful"  job  performance 
being  attained  by  individuals  at  various  aptitude  levels.  Probabilities  are 
computed  using  the  predictor  and  criterion  scores  of  some  known  cohort; 
thus,  it  is  assumed  that  predictor-criterion  relationships  will  remain  con¬ 
stant  with  future  cohons.  Various  proportions  of  individuals  meeting  a 
range  of  ASVAB  minimum  scores  are  related  to  probabilities  of  success  on 
the  job,  as  illustrated  in  Figure  3. 

The  expectancy  chart  method  seems  appropriate  for  an  Air  Force 
goal  such  as  the  following:  "to  determine  the  relationship  between  differ¬ 
ent  levels  of  ASVAB  scores  and  different  levels  of  job  performance  to 
justify  enlistment  standards  in  terms  of  impact  on  mission  capability."  For 
example,  expectancy  charts  would  provide  the  basis  for  an  argument  to  the 
effect  that  dropping  a  minimum  aptitude  score  from  85  to  70  would  mean 
a  10%  decrease  in  personnel  who  can  meet  minimal  levels  of  job  pro¬ 
ficiency.  A  required  input  is  thus  criterion-referenced  job  performance 
measures.  For  this  method,  the  required  inputs,  the  computational  pro¬ 
cesses  to  be  used,  and  the  resultant  outputs  are  as  follows: 

Inputs  Required 

1.  ASVAB  scores  for  known  cohort  of  job  incumbents; 

2.  For  same  incumbents,  job  performance  scores  that  reflect 
extent  of  job  mastery;  and 
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Group 

Minimum 

Score 

Percent  Who  Demonstrate  Minimal 

Level  of  Competency 

Best  20% 

85 

90 

Best  40% 

70 

80 

Best  60% 

53 

70 

Best  80% 

40 

60 

All 

25 

50 

Figure  3.  Institutional  Expectancy  Chart  Illustrating  the  Likelihood  of 
Successful  Criterion  Performance  at  Different  Levels  of 
Predictor  Scores. 


3.  Minimal  job  mastery  level  set  by  and  acceptable  to  policy 
makers. 

Computational  Processes 

1.  Compute  validity  coefficient  between  ASVAB  (predictor)  scores 
and  job  performance  (criterion)  scores; 

2.  Map  the  various  levels  of  ASVAB  performance  (top  20%,  40%, 
etc.)  onto  a  corresponding  set  of  minimum  ASVAB  scores;  and 

3.  Maximize  force  quality  by  estimating  likelihood  of  successful 
job  performance  (according  to  prespecified  minimally 
acceptable  standard)  from  any  given  level  of  aptitude. 

Outputs 


1 .  Probabilities  of  minimally  acceptable  job  performance 
associated  with  various  aptitude  levels;  and 


2. 


Range  of  potential  aptitude  cutoff  scores  and  associated 
impact  on  mission  due  to  adjustment  in  selection  standards. 


Evaluation  Results 

The  required  inputs  for  expectancy  charts  constitute  the  principal 
limitation  for  this  method.  Meaningful  output  depends  on  Job  performance 
measures  that  reflect  extent  of  job  mastery  (or  proficiency).  There  are  also 
several  assumptions  underlying  the  required  inputs  that  are  somewhat 
restrictive.  First,  the  relationship  between  predictor  and  criterion  scores  is 
assumed  to  be  constant  over  time.  This  assumption  could  be  periodically 
tested  in  revalidation  studies,  however.  Secondly,  a  minimally  acceptable 
level  of  job  mastery  is  assumed  as  a  standard  of  performance.  If  policy 
makers  are  reluctant  to  set  such  a  standard,  various  levels  could  be  model¬ 
ed  (as  in  Cascio  examples)  for  purposes  of  demonstrating  the  expectancy 
chart  technique.  Finally,  cost  factors  as  inputs  are  missing,  thereby  limit¬ 
ing  the  information  value  of  results;  however,  cost  factors  are  not  implied 
in  the  purpose  this  method  is  intended  to  serve.  With  respect  to  the  other 
criteria: 

1.  The  output  serves  the  stated  purpose; 

2.  The  results  are  easily  interpretable;  and 

3.  The  computational  requirements  are  reasonable. 


Break-Even  Analysis 


Description 

Break-even  analysis  was  the  second  method  described  by  Cascio 
(1987)  and  in  this  method  he  incorporated  cost  parameters.  The  basic 
objective  of  the  method  is  to  determine  the  minimum  aptitude  level  (i.  e., 
enlistment  standard)  that  will  allow  recovery  of  the  dollars  invested  in 
recruiting,  selecting,  inducting,  training,  and  monetarily  reimbursing  a 
specified  quota  of  personnel  until  minimally  acceptable  job  performance  is 
achieved.  This  method  is  a  form  of  the  ROI  approach. 

The  method  uses  the  general  utility  equation  which  has  the  following 
general  parameters:  costs  of  developing  a  recruit  into  a  minimally  satis¬ 
factory  job  performer  or  costs  associated  with  some  specified  time  period 


(C);  the  length  of  time  to  be  allowed  to  recover  investment  costs  (T);  the 
value  of  job  performance  expressed  in  dollar  terms  (SDy );  estimated  num¬ 
ber  of  new  recruits  required  in  some  future  time  period  (N);  the  correla¬ 
tion  of  predictor  and  criterion  scores  for  a  known  cohort  of  job  incumbents 
(r^y);  and  the  average  standard  score  (Z)  on  the  aptitude  test  required  to 
break  even  over  the  specified  time  period-  In  the  general  utility  equation, 
one  solves  for  A.  U  as  a  payoff  value.  In  this  application  A  U  is  set  to  0 
dollars  and  the  equation  is  solved  for  Z  to  derive  a  minimum  break-even 
aptitude  score. 

The  equation  is  defined  as: 

AU  =  N*T*rxy*SDy  *Z-N*C 

A  U  =  $0.00;  the  point  at  which  gains  due  to  improved  job  perform¬ 
ance  are  exactly  equal  to  investment  needed  to  reach  minimal 
competence 

N  =  the  specified  quota  of  new  recruits  in  some  future  time 
period 

T  =  the  length  of  time  available  to  recover  investment  (Cascio 
proposes  either  the  first  term  of  military  service  or  the 
average  tenure  of  personnel) 

r^y  =  the  validity  coefficient  of  aptitude  and  job  performance 
measures 

SDy  =  the  standard  deviation  of  job  performance  in  dollars 

Z  =  the  average  standard  score  on  the  selection  procedure 

required  to  break  even  over  the  time  period  (T)  specified. 

(Z  =  “x  /SR  where  >  is  the  ordinate  of  the  normal  curve 
corresponding  to  the  selection  ratio  [SR]  used) 

C  =  the  total  amount  of  money  invested  in  each  recruit  in  order  to 
bring  that  recruit  up  to  a  level  of  minimally  acceptable  job 
performance  or  to  complete  some  specified  time  period  of 
enlistment 

A  numerical  example  to  illustrate  the  use  of  the  equation  was  given 
by  Cascio  (1987,  p.  32-33)  where  N  =  10,000  recruits;  T  =  4  years;  r^y  = 
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0.40;  SDy  =  $15,000;  and  C  =  $20,000.  Solving  the  equation  yields  Z  =  0.83 
and  a  selection  ratio  of  0.20.  Therefore,  the  "solution"  is  to  use  a  cutoff 
score  which  would  identify  the  top  20%  of  the  examinees. 

Break-even  analysis  seems  appropriate  for  an  Air  Force  goal  such  as: 
"to  use  the  relationships  between  ASVAB  and  job  performance  scores  to 
derive  enlistment  standards  that  ensure  the  Air  Force's  investment  in 
personnel  is  recouped  during  their  enlistment."  For  example,  break-even 
analysis  would  provide  the  basis  for  an  argument  to  the  effect  that  a  given 
enlistment  standard  must  be  met  if  the  goal  is  to  get  a  payback  (in  terms 
of  airman  performance  contribution)  that  is  equivalent  to  the  Air  Force’s 
dollar  investment  in  the  airman  at  the  point  of  separation. 

For  break-even  analysis,  the  inputs,  computational  process,  and 
outputs  are  as  follows: 

Inputs  Required 

1.  ASVAB  and  job  performance  measurement  scores  for  cohort  of 
job  incumbents  (can  be  either  norm-referenced  or  criterion- 
referenced  tests); 

2.  rxj,:  the  validity  coefficient  of  aptitude  and  job  performance 
measures  in  a  known  cohort  of  job  incumbents,  corrected  for 
criterion  unreliability  and  range  restriction; 

3.  C:  all  costs  associa*'ed  with  recruiting,  induction  and  processing, 
initial  and  advanced  raining  (where  applicable),  and  salary 
and  other  benefits  during  Air  Force  tenure  or  during  period 
needed  for  minimal  proficiency  to  be  reached.  If  the  break¬ 
even  point  is  to  be  associated  with  reaching  minimal  pro¬ 
ficiency  as  opposed  to  completing  some  period  of  enlistment, 
then  a  minimal  proficiency  level  must  be  set  by  and  acceptable 
to  policy  makers; 

4.  T:  specfied  time  for  recovery  costs; 

5.  SDy  to  use  as  a  metric  to  express  the  worth  of  an  airman's 
contribution  to  the  Air  Force  during  a  specified  time  period; 
and 


6. 


N:  required  number  of  personnel. 


Computational  Process 

1.  Solve  for  Z  to  derive  a  minimum  aptitude  score  to  achieve  a 
"break-even  "  return  on  investment. 

Outputs 

1.  An  aptitude  cutoff  score  that  will  allow  airman  costs  and  per¬ 
formance  payback  during  enlistment  to  be  equivalent;  and 

2.  Byproduct  outputs  such  as  cost  figures  used  as  inputs. 
Evaluation  Results 

As  with  expectancy  charts,  the  required  inputs  for  break-even 
analysis  constitute  the  principal  limitation  for  this  method.  The  funda¬ 
mental  goal  of  making  costs  and  benefits  equivalent  in  dollar  terms 
assumes  that  airman  job  performance  can  be  meaningfully  represented  in 
terms  of  dollar  value  to  the  organization  (usually  expressed  as  salary). 
Although  there  has  been  some  work  done  to  derive  more  appropriate 
indicators  of  the  value  of  job  performance  in  military  contexts,  estimating 
performance  worth  remains  a  problematic  issue  for  any  cost-benefit 
method  that  uses  a  dollar  metric. 

Regarding  other  input  variables,  it  is  possible  to  implement  the 
break-even  approach  using  norm-referenced  job  performance  test  scores. 
The  resulting  cut-score  output  can  be  interpreted  only  in  terms  of  spec¬ 
ifying  the  level  of  aptitude  needed  for  the  Air  Force  to  recoup  its  invest¬ 
ment  costs  for  a  recruit.  With  norm-referenced  criterion  measures,  the 
derived  cut-score  does  not  provide  any  information  regarding  the  selec¬ 
tion  standard  needed  to  achieve  certain  levels  of  quality  in  job  perform¬ 
ance. 


With  respect  to  other  model  evaluation  criteria: 

1.  The  output  serves  the  stated  purpose; 

2.  The  results  are  easily  interpretable;  and 

3.  The  computational  requirements  are  reasonable,  assuming 
meaningful  measures  of  SDy  (or  its  equivalent)  can  be 
established. 
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Time  to  Proficiency  ("TTP^  Model 


Description 

The  time  to  proficiency  approach  (Carpenter  &  Monaco,  1987)  uses  a 
productivity  measure  to  establish  the  value  of  performance  output  rather 
than  using  a  dollar  metric  as  seen  in  Cascio's  break-even  approach.  (Cascio 
[1987]  also  proposed  as  an  alternative  to  SDy,  SDp,  which  is  defined  as  the 
standard  deviation  of  output  expressed  as  a  percentage  of  mean  output. 

The  TTP  approach  is  conceptually  similar  to  Cascio’s  SDp.)  The  objective  is 
to  determine  the  aptitude  cutoff  score  that  minimizes  cost  per  productive 
unit  of  performance. 

The  approach  features  the  notion  of  "time  to  proficiency’’—  defined 
as  the  length  of  time  it  takes  to  bring  people  with  different  attributes  (e.  g., 
mental  aptitude)  to  targeted  levels  of  proficiency  (i.  e.,  productivity). 

Given  the  basic  parameters  of  productivity,  costs,  and  aptitude  indices,  this 
approach  seems  appropriate  to  an  Air  Force  objective  such  as  linking  apti¬ 
tude  and  performance  (which,  in  the  TTP  model,  is  represented  as  product¬ 
ive  capacity)  to  minimize  costs  while  fielding  a  capable  enlisted  force. 

Stated  differently,  the  TTP  approach  finds  "those  levels  of  aptitude  re¬ 
quirements  that  best  balance  work  force  capabilities  and  costs"  (Carpenter 
&  Monaco,  1987,  p.  1).  Characterized  in  this  manner,  TTP  has  an  ROI 
objective  similar  to  break-even  analysis.  The  major  difference  is  in  the 
metric  used  to  represent  capabilities  of  the  force.  Break-even  analysis 
uses  a  dollar  metric  (SDy)  whereas  TTP  represents  capability  as  work  units 
com-pleted  per  unit  of  time. 

To  accomplish  the  objective  of  minimizing  cost/productive  unit,  the 
TTP  approach  examines  the  functional  and  logical  relationships  of  four 
components;  aptitude,  productivity  (work  output),  cost,  and  attrition.  In 
overview  terms,  productivity  is  represented  as  a  function  of  performance 
time,  experience,  and  aptitude.  Distributions  of  aptitude  scores  are  used  so 
that  probability  density  functions  can  be  computed  for  subpopulations  of 
airmen  defined  by  level  of  aptitude.  Costs  are  represented  as  recruiting, 
training,  and  salary  costs  through  the  first  term.  Attrition  is  represented 
as  the  likelihool  of  remaining  in  the  service  for  each  month  in  the  first 
term  of  enlistment  as  a  function  of  aptitude. 
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Basically,  production  and  cost  are  accumulated  over  the  first  term 
and  adjusted  for  attrition  to  yield  expected  first-term  productive  capacity 
(the  inverse  of  performance  time)  and  expected  first-term  cost.  The  ratio 
of  expected  cost  to  expected  productive  capacity  is  then  computed  for 
aptitude  subgroups  to  determine  the  subgroups  with  the  lowest  average 
cost  per  productive  unit  for  the  first  term.  The  mathematical  representa¬ 
tions  of  the  four  models  that  are  integrated  to  form  the  TTP  model  are 
described  next. 

The  productive  capacity  model  is  defined  as 
P  =  1/  (1  +  exp  [  -  bo  -  bi  XI  -  b2  X2]  ) 

where 

P  =  productive  capacity 

xi  =  experience  (months  in  -AFS) 

X2  =  mental  aptitude  index  (e.g.,  ASVAB  electronics  score) 
bg,  bi,  and  b2  are  parameters  to  be  estimated. 

The  productive  capacity,  P,  is  defined  as  t*/t,  where  t*  is  an  estimate 
of  the  fastest  possible  performance  time  in  which  a  defined  unit  of  work 
can  be  completed  at  an  acceptable  level  of  quality  (based  on  supervisors' 
ratings  of  subordinates'  relative  required  times  for  acceptable  task  com¬ 
pletion).  For  any  individual  worker,  t  is  the  time  required  for  that  worker 
to  satisfactorily  complete  the  defined  unit  of  work.  For  the  individual,  1/t 
is  the  number  of  work  units  he  or  she  can  complete  per  unit  of  time.  Thus: 

P  =  t*/t  =  (1/t)  /  (1/t*)  is  the  proportion  of  maximum  produc¬ 
tivity  that  the  individual  has  obtained  (Carpenter  &  Monaco, 
1987).  Further,  this  model  assumes  that  airmen  with  higher 
aptitudes  are  more  productive  than  are  airmen  with  lower 
aptitudes  and  that  productivity  increases  with  experience. 

The  attrition  model  is  defined  as; 


r  (x,  t)  =  bo  +  bj  In  [(t  +  s  (x))  /  (48  -  t)]  +  b4  x 

where  r(x,t)  is  the  probability  of  an  airman  with  an  aptitude 
index  of  x  remaining  in  service  after  t  months  (t  =  1  to  48) 


X  =  mental  aptitude  index  (e.g.,  ASVAB  electronics  score  or  E- 
score) 

t  =  time  in  service  (months) 

s  (x)  =  exp  (b2  +  b3  x) 

bg,  bj,  b2,  b3,  b4  are  parameters  to  be  estimated. 

Once  estimated,  the  above  productive  capacity  and  attrition  models 
are  integrated  into  a  meaningful  objective  function.  To  develop  this 
function,  it  is  helpful  to  first  consider  two  intermediate  functions  — 
expected  productive  capacity  and  expected  cost.  The  ratio  of  expected  cost 
to  expected  productive  capacity  provides  the  desired  index  of  cost  per 
productive  units  of  performance.  Expected  first-term  productive  capacity 
for  an  individual  with  a  given  E-score  of  x  can  be  described  by  the 
following  equation; 

P(x)  =  2^  r(x,t)  p(x,t) 

t*  0 

where  P  (x)  =  Expected  first-term  productive  capacity  for  an 
individual  with  E-score  of  x 

X  =  E-score 

t  =  time  in  service  (months) 

r(x,t)  =  probability  that  an  individual  with  E-score  of  x  is 
still  in  service  after  t  months 


p(x,t)  =  productive  capacity  for  an  individual  with  an 
E-score  of  x  and  t  months  of  experience. 


Expected  first-term  cost  for  an  individual  with  a  given  E-score  can  be 
described  by: 


-fit 


C(x)  =  2, 

■t’C 


Expected  first-term  cost  for  an  individual  with  an 
E-score  t)f  x 


Where  C(x) 


X 


E-score 


t  =  time  in  service  (months) 

r(x,t)  =  probability  that  an  individual  with  E-score  of  x  is 
still  in  service  after  t  months 

c(x,t)  =  cost  to  the  Air  Force  of  an  individual  with  E-score  x 
in  month  t. 

The  ratio  C(x)/P(x)  represents  the  expected  cost  per  productive  unit 
over  the  first  term.  By  minimizing  this  ratio  with  respect  to  x  (E-score), 
one  can  determine  the  optimum  E-score  to  minimize  cost  per  productive 
unit. 

At  this  stage  of  the  minimization  problem,  the  formulas  imply  that 
all  new  recruits  should  be  selected  with  an  E-score  equal  to  the  optimum 
value  of  X.  It  would,  of  course,  be  impractical  to  impose  such  a  constraint 
on  the  selection  of  airmen  for  a  given  AFS.  The  distribution  of  aptitude 
scores  of  the  pool  from  which  new  recruits  are  selected  must  be  consider¬ 
ed.  If  f(x)  is  the  probability  density  function  of  E-scores  for  the  population 
of  potential  recruits,  then 

fm(x)  =  f(x)/(l-  ^  f(i))  is  the  conditional  probability  density  func- 

tion  for  the  population  of  potential  recruits  with  an  E-score  of 
at  least  m.  Productive  capacity  and  cost  can  then  be  modeled 
as  functions  of  a  minimum  allowable  E-score  (as  opposed  to  an 
optimum). 

E  [P(m)]  =  21  fra(x)  P(x) 

E  [C(m)]  =  Z  fm(x)  C(x) 

•X  :^ry» 

Where;  m  is  the  minimum  allowable  E-score 


E[P(m)]  is  the  expected  (average)  first-term  productive  capaci¬ 
ty  for  the  subpopulation  of  potential  recruits  with  E-scores  of 
at  least  m 

E  [C(m)]  is  the  expected  (average)  first-term  cost  for  that 
subpopulation 
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fni(x)  is  the  conditional  probability  density  function  of  E-scores 
for  the  subpopulation  of  potential  recruits  with  E-scores  of  at 
least  m 

P(x)  and  C(x)  are  as  defined  earlier. 

By  minimizing  the  ratio  E[C(m)]/E[P(m)]  with  respect  to  m,  one  can 
find  the  optimum  minimum  E-score  standard  for  a  given  distribution  of  E- 
scores  among  a  recruiting  pool. 

The  inputs,  computational  processes,  and  outputs  for  the  TTP  Model 
are  as  follows: 

Inputs  Required 

o  Productive  capacity  model: 

1.  t*:  estimate  of  smallest  possible  time  interval  in  which  a 

defined  unit  of  work  can  be  satisfactorily  completed  (based  on 
supervisor  ratings); 

2.  t:  time  required  for  a  given  worker  to  satisfactorily 

complete  the  defined  unit  of  work; 

3.  xi:  months'  experience  in  the  AFSC  (by  worker); 

4.  X2:  aptitude  score  (by  worker);  and 

5.  Distribution  of  aptitude  scores  of  the  pool  of  possible  recruits. 

o  Attrition  model  (only  those  inputs  unique  to  this  model  are 
listed): 

6.  r  (x,  t):  probability  of  an  airman  with  an  aptitude  of  x 
remaining  in  the  service  after  t  months  (t  =  1  to  48). 

0  Cost  model 

7.  C:  first-term  costs  using  average  recruiting  and  training  costs 
and  military  pay. 
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1.  Estimate  productive  capacity  as  the  inverse  of  performance 
time; 

2.  Estimate  expected  productive  capacity  by  adjusting  productive 
capacity  for  attrition  effects;  and 

3.  Minimize  cost  per  productive  unit  of  performance  (i.  e.,  benefit) 
by  computing  the  ratio  of  expected  cost  to  expected  productive 
capacity. 

Outputs 

1.  An  optimum  aptitude  standard  that  minimizes  cost  per  unit  of 
production;  and 

2.  Byproduct  outputs  such  as 

a.  Cost  figures  used  as  inputs, 

b.  Expected  productive  capacity  of  an  individual  with  a 
given  aptitude  score  at  a  given  point  in  his/her  career, 
and 

c.  Predicted  attrition  rate  by  aptitude  score. 

An  optimal  solution  using  the  TTP  model  is  illustrated  in  Figure  4 
(from  Carpenter  &  Monaco,  1987),  where  the  computed  expected  cost  per 
productive  unit  (vertical  axis)  is  shown  for  subpopulations  defined  by 
different  electronics  score  cutoffs  (horizontal  axis).  In  this  example,  the 
lowest  average  cost  is  associated  with  an  electronics  score  cutoff  of  90  (bar 
at  far  right).  Under  a  different  cost  model,  the  solution  might  change.  For 
example,  using  a  different  cost  model  reflecting  increased  recruiting  costs 
and  military  pay  for  higher  aptitude  groups  would  influence  the  solution  in 
the  direction  of  choosing  a  lower  aptitude  subpopulation.  An  example  of 
such  a  solution  is  shown  in  Figure  5  (from  Carpenter  &  Monaco,  1987), 
where  the  costs  for  the  highest  subpopulation  (i.  e.,  cutoff  of  90)  have  been 
increased  by  9%.  As  a  result,  the  lowest  average  cost  per  productive  unit 
in  this  case  is  obtained  for  a  lower  subgroup;  namely,  the  group  with  a 
cutoff  of  80. 


Average  2.5 
Cost  Per 
Productive  2.0 
Unit  In 

Thousands  1.5 


Electronics  Score  Cutoff 


k  Average  Cost  Per  Productive  Unit. 
(Carpenter  &  Monaco,  1987) 


Electronics  Score  Cutoff 


Figure  5 


Average  Cost  Per  Productive  Unit  with  9% 
Cost  Increase  (Carpenter  &  Monaco,  1987) 


Oualitv/Ohtainahilitv  Of  Required  Inputs.  A  potential  benefit 
associated  with  the  TTP  model  is  that  it  does  not  require  criterion- 
referenced  job  performance  measures  as  inputs,  as  contrasted  to  most 
other  approaches  (break-even  analysis  is  the  other  exception).  In  fact,  this 
approach  does  not  use  available  JSP  job  performance  measures  directly; 
however,  the  Walk-Through  Performance  Test  was  used  to  assess  the  TTP 
as  a  surrogate.  In  the  TTP  procedure,  performance  is  estimated  by  time  to 
proficiency  ratings  provided  by  supervisors.  The  collection  of  the  job 
performance  inputs  for  this  approach  therefore  requires  a  moderate 
degree  of  effort.  A  documented  procedure  exists  for  collecting  TTP  data 
and  developing  surrogate  job  performance  measures;  however,  to  date  this 
has  been  accomplished  on  only  one  AFS.  Required  inputs  to  the  attrition 
and  cost  models  are  reasonably  obtainable,  although  some  recruiting  costs 
are  difficult  to  estimate. 

There  are  additional  important  caveats  to  be  stated  regarding  any 
form  of  personnel  ratings  provided  by  supervisors.  As  Wigdor  and  Green 
(1986)  noted,  supervisor  ratings  are  known  to  be  contaminated  by 
personal  attitudes  such  as  personality  preferences.  On  the  plus  side, 
however.  Carpenter  and  Monaco  (1987)  established  limited  validity  for  the 
supervisor  ratings  obtained  in  their  study  of  one  AFS.  In  addition,  other 
measures  of  time  to  perform  would  work  equally  well  in  the  TTP  approach 
(e.g.,  the  JSP's  hands-on  performance  time). 

A  final  comment  is  offered  on  the  quality  of  TTP  indices:  They  are 
norm-referenced  in  the  sense  that  time  requirements  (for  completion)  are 
represented  for  a  group  of  workers  in  relation  to  one  another.  That  means 
that  the  production  or  productive  capacity  indicators  depict  proficiency  in 
a  relative  sense,  not  in  an  absolute  sense  that  would  explain  whether  the 
units  of  production  that  are  estimated  represent  95%  of  the  job  or  50%,  for 
example. 

Other  Evaluative  Criteria.  In  terms  of  computational  efficiency  and 
flexibility,  the  TTP  approach  has  been  satisfactorily  tested  on  an  Air  Force 
data  set  to  verify  the  ’'workability"  of  the  modeling  process.  In  terms  of 
interpretability  of  the  output.  Figures  4  and  5  illustrate  the  comprehen¬ 
sible  way  in  which  it  is  possible  to  represent  findings,  even  though  the 
model  building  leading  up  to  the  results  is  involved  and  intricate. 

Finally,  in  terms  of  the  suitability  of  the  output,  given  the  stated  goal, 
the  TTP  approach  does  allow  one  to  determine  "those  levels  of  aptitude 


requirements  that  best  balance  work  force  capabilities  and  costs” 
(Carpenter  &  Monaco,  1987,  p.  1).  The  work  force  capabilities  that  are 
utilized  are  norm-referenced  and  therefore  they  assume  that  present 
members  of  the  enlisted  force  meet  implicit  standards  of  competence, 
making  it  unnecessary  to  represent  performance  in  terms  of  extent  of  job 
mastery.  Further,  this  approach  does  not  explain  the  empirical  linkages 
between  concrete  behavioral  job  requirements  and  aptitude  constructs,  as 
favored  by  the  National  Academy  of  Sciences  Committee.  However,  that 
dimension  is  not  part  of  the  goal  to  which  this  approach  responds. 


The  Rand  Model 


Description 

By  far  the  most  extensively  documented  linkage  model  is  the  Rand 
cost/  performance  tradeoff  model  (Armor,  Fernandez,  Bers,  & 

Schwartzbach,  1982).  The  basic  objective  of  this  method  is  to  derive 
optimal  enlistment  standards  and  ability  mixes  (among  personnel)  by 
calculating  force-cost  figures  for  a  given  level  of  job  performance.  As  a 
cost  effectiveness  approach,  it  serves  the  purpose  of  determining  the 
"enlistment  standards  for  a  given  military  job  that  minimize  the  cost  of 
achieving  given  levels  of  job  performance"  (Armor  et  al.,  1982,  p.  14).  In 
treating  "levels  of  performance,"  the  Rand  model  features  the  notion  of 
qualified  man-months  (OMMs).  It  is  the  period  of  enlistment  during  which 
a  performer  functions  at  a  qualified  (versus  unqualified)  level.  The  mile¬ 
stone  that  distinguishes  between  the  two  levels  is  the  meeting  of  a  mini¬ 
mal  standard  of  proficiency  on  a  job  performance  test  such  as  the  Army's 
Skill  Qualification  Test  (SQT).  Months  in  service  after  passing  a  test  such  as 
the  SQT  are  regarded  as  QMMs. 

To  accomplish  the  stated  objectives,  the  Rand  approach  integrates  a 
performance  model  and  a  cost  model  to  yield  force-cost  estimates  (as 
inputs)  to  produce  the  desired  output;  namely,  ability  mixes  that  mini¬ 
mize  force  cost  per  QMM.  In  the  performance  model,  a  minimum  standard 
is  set  for  a  job  performance  test  such  as  the  Army's  SQT.  As  noted  earlier, 
months  in  service  after  passing  the  SQT  are  regarded  as  QMMs. 

In  order  to  ensure  that  end-strength  or  necessary  manpower  levels 
are  met  in  the  calculation  of  QMMs,  a  constraint  on  the  model  is  typically 
invoked  whereby  a  predetermined  number  of  retained  man-months 


(RMMs)  must  be  met.  The  number  of  RMMs  essentially  reflects  needed 
manpower  to  fill  established  quotas. 

To  summarize,  numbers  of  QMMs  yielded  by  various  ability  mixes 
are  modeled  as  a  function  of  personnel  entry  characteristics  such  as 
aptitude  and  (for  Army  applicants)  high  school  completion.  The  cost  model 
then  estimates  the  cost  of  recruiting  and  maintaining  various  levels  of  a 
qualified  force  in  terms  of  ability  mixes  at  time  of  entry.  The  mathema¬ 
tical  representations  of  the  models  that  are  integrated  to  produce  the 
desired  output  are  described  next.  Since  the  Rand  approach  has  been 
developed  and  tested  on  Army  populations,  discussions  are  necessarily  in 
that  context. 

The  performance  model  produces  as  output  QMMs.  A  simple 
accounting  function  relates  QMMs  to  two  major  enlistment  characteristics: 
high  school  completion  and  aptitude  level  (expressed  by  Armor  et  al. 

[1982],  as  enlistment  categories  I  through  IV).  The  remaining  parameters 
are  functions  of  those  entry  characteristics  found  to  predict  basic  training 
completion  (B),  advanced  training  completion  (T),  retention  (R),  and  SQT 
performance  (S). 

QMMs  are  computed  using  the  following  expression: 

QMMs  =  AXfPiBiTi2I(Ri,Si,)l  (1) 

U  t 

where 

i  =  category  of  entry  characteristics  (e.  g.,  high  school  graduate  in 
enlistment  category  II) 

t  =  time  in  service  in  months 

A  =  number  of  accessions 

Pj  =  proportion  of  accessions  in  category  i 

Bi=  proportion  of  category  i  accessions  completing  basic  training 

Tj  =  proportion  of  category  i  basic  graduates  completing  advanced 
training 

Rit  =  proportion  of  category  i  advanced  graduates  completing  t 
months  of  service 


Sit  =  proportion  of  category  i  retained  accessions  able  to  pass  the 
SQT  at  month  t 

The  objective  of  the  Rand's  cost  model  is  to  impose  cost  fqures  on 
the  previously  defined  performance  model  to  make  the  output  more 
meaningful;  i.  e.,  to  establish  the  cost  of  recruiting  and  maintaining  a  given 
qualified  force  (as  implied  in  equation  1).  Together  the  performance  and 
cost  models  can  be  used  to  investigate  ability  mixes  that  minimize  force 
costs  per  QMM.  In  this  sense  the  Rand  and  TTP  approaches  are  similar 
since  they  both  seek  "minimal  costs  for  productivity  attained"  (here  pro¬ 
ductivity  is  months  of  service  at  a  qualified  level;  with  TTP,  productivity  is 
work  units  completed  per  unit  of  lime). 

Force  costs  for  a  given  ability  mix  fall  into  the  following  categories 
according  to  Armor  et  al.  (1982,  p.  18):  (a)  the  cost  of  recruiting,  including 

advertising  and  special  enlistment  benefits  and  bonuses  offered  to  high 
ability  personnel  (Ca);  (b)  cost  of  initial  entry  and  basic  training  (Cb);  (c) 
cost  of  advanced  training  in  a  specific  job  (Cx);  and  (d)  cost  of  the  retained 
force  in  the  post-training  period,  including  pay,  allowances,  and  base 
support  activities  (medical,  commissary,  recreation,  etc.)  (Cr).  Of  these,  the 
cost  of  recruiting  is  considered  the  most  difficult  to  estimate.  Total  cost  of 
recruiting  and  maintenance  is  merely  the  sum  of  these  four  component 
costs: 


C  =  Ca  +  Cb  +  Ct  +  Cr.  (2) 

Two  variations  of  integrating  the  performance  and  cost  models  -- 
RAND-1  and  RAND-2  -  have  been  proposed  by  Armor  et  al.  (1982).  The 
objective  being  modeled  and  the  constraints  invoked  are  different  in  each. 

In  RAND-1,  RMMs  are  constrained  to  a  fixed  level  (to  meet  man¬ 
power  quotas)  and  the  cost  per  QMM  is  minimized.  The  formula  for  RRMs 
is  shown  below.  The  job  performance  SQT  parameter  has  been  removed; 
otherwise  the  formula  is  the  same  as  equation  1: 

RMMs  =  AXlPi  Bi  TiX(Rit)].  (3) 

U  t 

The  RAND-1  model  for  cost  minimization  as  constrained  by  fixed 
manpower  requirements  (represented  by  RMMs)  is  illustrated  below  (from 
May  &  Mayberry,  1986).  For  purposes  of  illustration,  personnel  are  divid¬ 
ed  into  two  groups  —  high  quality  (h)  and  low  quality  (1). 
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C  (  Mh  (s).  Ml  (s)  ) 


Minimize  —————— 

Q  (  Mh  (s).  Ml  (s)  ) 

Subject  to  Mh  (s)  +  Mi  (s)  =  RMM* 

where 


C=  total  cost 
Q  =  number  of  QMMs 

Mh=  number  of  RMMs  for  high-quality  recruits 
Ml  =  number  of  RMMs  for  low-quality  recruits 
s  =  aptitude  cutoff  score 
RMM*  =  fixed  number  of  RMMs 

In  RAND-2,  the  number  of  QMMs  is  held  constant  and  total  cost  is 
minimized,  but  force  size  is  allowed  to  vary.  This  variation  greatly  limits 
the  utility  of  RAND-2. 

The  required  inputs,  computational  processes,  and  outputs  for  the 
RAND-1  and  RAND-2  models  are  as  follows: 


Inputs  Required 

•  Performance  model 


1. 

A: 

2. 

P; 

3. 

B: 

4. 

T: 

total  number  of  accessions  to  be  considered; 

proportions  of  accessions  in  given  aptitude  categories; 

proportions  of  accessions  completing  basic  training  (by 
aptitude  category); 

proportions  of  basic  training  graduates  completing  tech¬ 
nical  training  (by  aptitude  category): 
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R;  months  of  service  (t)  completed  by  technical  training 
graduates  (by  mental  category);  and 

S;  proportion  of  retained  accessions  able  to  pass  a  job 

performance  test  such  as  the  SQT  (by  aptitude  category 
and  month  of  SQT  passed)  --  includes  mir  .nally 
acceptable  job  performance  test  score. 

Cost  model 

C  sum  of  force  costs  to  include  recruiting  (0^.),  initial  entry 
and  basic  training  (Cb),  technical  training  (Ct),  and  force 
maintenance  (Cr). 

mat  Processes 

Estimate  quality  of  performance  for  aptitude  subgroups  in 
terms  of.  the  number  of  months  an  individual  performs  at  a 
"qualified"  level  (meaning  a  minimal  proficiency  standard  has 
been  met); 

Ensure  fixed  manpower  requirements  are  met  (in  RAND-1)  by 
constraining  cost  minimization  computations  so  that  necessary 
total  man-months  are  achieved;  and 

Minimize  cost  per  qualified  unit  of  performance  (i.e.,  benefit). 


An  optimum  aptitude  standard  (cutoff  score)  and  associated 
aptitude  mixes  that  minimize  cost  per  qualified  unit  of 
performance; 

Numbers  of  qualified  personnel  achieved  with  optimal  stand¬ 
ard  and  associated  costs;  and 

Byproduct  outputs  such  as 

a.  Alternative  standards  and  associated  aptitude  mixes, 
numbers  of  personnel,  and  associated  costs;  and 

Cost  figures  used  as  inputs. 
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b. 


An  example  of  the  type  of  information  obtained  from  the  processing 
of  the  Rand  model  using  Army  data  is  displayed  in  Table  1.  Note  that  as 
the  cutoff  score  is  moved  from  a  low  value  of  70  upward  to  a  high  value  of 
115  the  ability  mix  with  regard  to  education  and  mental  category  changes 
considerably.  For  example,  the  number  of  high  sc^.'^ol  graduates  in  cate¬ 
gories  I-IIIA  is  8,513  at  the  115  cutoff  level  versus  2,902  at  the  70  cutoff 
level.  However,  costs  are  higher  at  the  115  level  (i.  e.,  $2,503  per  QMM 
versus  $2,260).  Examination  of  the  cost  per  QMM  column  reveals  the  mini¬ 
mum  to  be  located  at  a  cutoff  score  of  85.  Thus,  for  this  example,  the 
desired  cutoff  score  is  85. 

Table  1.  Rand  Model  Sample  Output 


Number 


Cutoff 

Score 

of 

Accessions 

HS 

I-fflA 

Non-HS 

I-IIIA 

HS 

IIIB-IV 

Non-HS 

ins-rv 

Cost  per 
QMM 

115-^ 

11,242 

8,513 

2,309 

257 

163 

2,503 

110-H 

11,331 

7,796 

2,689 

495 

351 

2,397 

105-t- 

11,510 

7,066 

2,917 

861 

666 

2,308 

100-h 

11,644 

6,434 

2,930 

1,364 

916 

2,247 

95+ 

11,757 

5,547 

2,822 

2,108 

1,280 

2,179 

90+ 

11,756 

4,732 

2,572 

2,965 

1,487 

2,142 

85+ 

11,728 

4,331 

2,404 

3,475 

1,519 

2,139 

80+ 

11,651 

3,747 

2,144 

4,252 

1,507 

2,153 

76+ 

11,580 

3,267 

1,897 

4,979 

1,437 

2,189 

70+ 

11,523 

2,902 

1,709 

5,529 

1,384 

2,260 
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Oualitv/Qbtainabilitv  of  Required  Inputs.  The  Rand  model  pursues 
the  same  objective  as  the  TTP  approach  and  to  some  extent  break-even 
analysis;  namely,  to  minimize  costs  in  achieving  a  qualified  work  force.  For 
all  three  models  the  difficulty  comes  in  defining  the  criteria  of  a  qualified 
force.  The  input  used  to  serve  this  purpose  in  the  Rand  model  is  a  stand¬ 
ard  on  a  competency-based  job  performance  test.  Such  a  criterion  mea¬ 
sure  is  a  problem  for  the  time  to  proficiency  approach  and  other  norm- 
referenced  testing  programs.  Presumably,  norm-referenced  tests  could  be 
used  and  arbitrary  cutoff  scores  examined  as  illustrated  in  Table  1;  how¬ 
ever,  outcomes  would  be  difficult  to  interpret  since  the  scores  would  not 
reveal  adequacy  of  performance  in  any  absolute  sense. 

There  are  several  other  factors  related  to  the  quality  of  Rand  input 
variables  that  should  be  noted  as  well.  First,  the  manner  in  which  QMMs 
are  computed  assumes  a  dichotomous  qualified  versus  unqualified  per¬ 
formance  variable.  This  is  undesirable  since  relative  performance  differ¬ 
ences  are  ignored,  and  the  model  thus  favors  the  recruiting  of  individuals 
who  score  just  above  the  cutoff  rather  than  those  higher  scoring  individ¬ 
uals  who  are  assumed  to  be  more  expensive  to  recruit.  A  continuous 
measure  of  job  performance  might  be  preferable  and  should  be  tested  with 
the  Rand  approach  using  Air  Force  data.  Some  preliminary  work  concern¬ 
ing  the  use  of  continuous  measures  for  Army  personnel  has  been  under¬ 
taken  by  Armor  (1985).  Secondly,  the  distribution  of  personnel  across 
aptitude  categories  in  the  cohort  used  in  the  analysis  affects  t!  outputs  of 
the  model.  This  so-called  "proportionality  assumption"  presupposes  that 
recruits  in  future  cohorts  will  be  similarly  distributed  so  that  ability  mixes 
will  remain  similar  to  the  "baseline"  accessions  group.  The  dependency  of 
the  Rand  optimization  procedures  on  the  structure  of  the  baseline  group  is 
illustrated  with  the  following  example.  Suppose  at  the  cutoff  score  of  75, 
four  hypothetical  groups  of  recruits  had  equal  numbers  or  proportions: 
Group  1  =  25%,  Group  2  =  25%,  and  Group  3  =  25%,  and  Group  4  =  25%. 
Suppose  further  that  by  increasing  the  cutoff  score  to  80,  ail  Group  4  per¬ 
sonnel  would  be  eliminated.  According  to  the  proportionality  -  assumption, 
the  new  distribution  of  recruits  would  be  Group  1  =  25/75  =  33  1/3%, 

Group  2  =  25/75  =  33  1/3%,  Group  3  =  25/75  =  33  1/3%,  and  Group  4  =  0%. 
Thus,  outcomes  are  dependent  upon  the  choice  of  the  baseline  group. 


Other  Evaluative  Criteria.  In  terms  of  computational  efficiency  and 
flexibility,  the  Rand  approach  has  been  tested  on  Army  data  and  appears 
workable.  The  use  of  a  continuous  (versus  dichotomous)  job  performance 
measure  may  be  a  useful  way  to  further  test  the  model’s  flexibility.  The 


model  building  process  is  somewhat  intricate  with  the  Rand  approach,  a 
condition  which  undoubtedly  results  in  greater  processing  time  when  com¬ 
pared  to  simpler  models  such  as  break-even  analysis.  However,  the  output 
data  are  rich  and  presumably  justify  the  requisite  processing  time. 

In  terms  of  interpretability  of  results.  Table  1  illustrates  a  satisfact¬ 
orily  comprehensible  array  of  findings.  Finally,  in  terms  of  the  suitability 
of  the  output  given  the  stated  goal,  the  results  do  reveal  costs  per  qualified 
units  of  performance.  The  limitations  revolve  around  defining  just  what 
those  qualified  units  are  and  how  the  potential  to  achieve  qualified  status 
is  distributed  among  a  population  of  recruits. 

Summary  of  Linkage  Approac hes 


In  the  following  table,  the  four  models  for  linking  aptimde  and  job 
performance  are  summarized  in  terms  of  the  evaluative  framework  that 
was  used  in  the  previous  sections:  goal  to  be  served  by  the  linkage  model, 
input/computational  processes/output,  and  model  strengths  and  weak¬ 
nesses.  The  reader  will  notice  a  basic  similarity  in  purpose  across  the  four 
approaches;  namely,  to  use  the  aptitude-performance  relationship  to  im¬ 
prove  decision  making  about  the  composition  of  the  enlisted  force.  What 
distinguishes  some  approaches  is  that  costs  are  factored  into  the  decision 
making. 


Table  2.  Summary  of  Linkage  Models 
Model:  Expectancy  Charts 


To  determine  probability  of  job  proficiency  given 
level  of  entry  aptitude 

To  determine  ASVAB-job  performance  relationships 
to  establish  impact  of  standards  on  mission 
capability 

Aptitude  scores 

Competency-based  job  performance  measures 

Minimal  competency  standard 


Maximize  force  quality  by  computing  probabilities 
of  job  proficiency  for  given  aptitude  levels 

Probabilities  of  success  for  a  given  aptitude  group 
Basis  for  justifying  enlistment  standards  in  terms  of 
mission  impact 

ios.:  Competency-based  tests  and  level  of  minimal 
competency  required 
Computational  demands  low 
Results  easily  interpretable  but  limited  in  useful¬ 
ness  since  cost  information  is  not  used 


Table  2.  (Continued) 
Model:  Break-Even  Analysis 


Objective: 

Air  Force  Goal: 

Primary  Input: 

Computational 
Processes 
Objective  Function: 

Primary  Output: 

Strengths/Limitations: 


To  determine  aptitude  levels  that  will  yield 
equivalent  personnel  costs  and  benefits 

To  use  ASVAB-job  performance  relationships  to 
establish  enlistment  standards  that  ensure 
return  on  investment  in  personnel 

Aptitude  scores 

Job  performance  scores  (either  norm-  or  criterion- 
referenced) 

Personnel  costs  expressed  in  dollars 

Personnel  benefits  expressed  in  dollars  (e.g.,  salary 
metric  such  as  SDy) 

Time  to  achieve  return  on  investment 


Make  return  on  investment  (i.e.,  benefits  to  cost 
ratio)  equivalent 

Aptitude  cutoff  scores  (enlistment  standards)  to 

achieve  equivalence  between  personnel  costs 
and  performance  benefits 

Basis  for  justifying  enlistment  standards  from 

return  on  investment  monetary  perspective 

Cost  information  allows  cost-benefit  tradeoff 
analysis 

Obtaining  a  dollar  metric  for  performance  benefits 
is  problematic 

Computational  demands  reasonable 

Results  interpretable  assuming  meaningful  dollar 
metric  for  performance  benefits  identified 
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Table  2.  (Continued) 
Model:  Time  to  Proficiency 


To  determine  aptitude  cutoff  that  minimizes  cost 
per  productive  unit  of  performance 

To  use  ASVAB-job  performance  relationships  to 

minimize  costs  while  fielding  a  capable  force; 
i.e.,  to  balance  force  capability  and  cost 

Aptitude  scores 

Supervisor  estimates  of  performance  expressed  as 
the  relative  time  to  produce  a  satisfactory 
unit  of  work 

Estimates  of  personnel  costs 

Attrition  rates  by  aptitude  group 


Minimize  cost  per  productive  unit  of  performance 
by  computing  ratio  of  expected  cost  to 
expected  productive  capacity  (adjusted  for 
attrition  effects) 

Aptitude  cutoff  scores  (enlistment  standards)  that 
minimize  cost  per  unit  of  production 
Basis  for  justifying  enlistment  standards  from  cost- 
benefit  perspective  that  does  not  require 
dollar  metric  to  represent  performance 
(benefits) 

TTP  avoids  dollar  metric  problem  (of  Break-Even 
Analysis)  but  substitutes  supervisory 
judgment  approach  that  may  have  biases 
Supervisor  ratings  are  norm-referenced;  thus, 

resultant  productivity  indices  are  relative  to 
other  performers,  not  job  content 
Computational  feasibility  of  model  previously 
demonstrated 

Results  interpretable  but  require  caveats 


Table  2.  (Concluded) 


Model;  Rand  Cost/Performance  Tradeoff  Model 


To  determine  aptitude  cutoff  and  ability  mixes  that 
minimize  cost  per  qualified  unit  of 
performance 

To  use  ASVAB-job  performance  relationships  to 
minimize  costs  while  fielding  a  maximally 
qualified  force 


Aptitude  scores 


rformance  measures 


Minimal  competency  standard 
Manpower  requirements  that  must  be  met 
Estimates  of  personnel  costs 


Minimize  cost  per  qualified  unit  of  performance 


Aptitude  cutoff  scores  and  associated  aptitude 

mixes  that  minimize  cost  per  qualified  unit  of 
performance 

Basis  for  justifying  enlistment  standards  from  cost- 
benefit  perspective  that  can  express  input  in 
terms  of  lowered  mission  capability 


ths/Limitations:  Competency-based  tests  and  level  of  minimal 

competency  required 

When  performance  scores  are  treated  as  dichoto¬ 
mous  variables,  selection  favors  personnel 
near  cutoff 

Rich  set  of  results  produced:  optimal  and  alterna¬ 
tive  enlistment  standards  and  associated 
ability  mixes 

Computational  feasibility  of  model  has  been 
demonstrated 


V.  CONCLUSIONS  AND  RECOMMENDATIONS 


Conclusions 


The  general  conclusions  to  be  drawn  from  the  previous  evaluation 
are  as  follows: 

1.  The  first  step  in  pursuing  the  linking  of  aptitude  to  job  per¬ 
formance  is  the  key  one.  The  Air  Force  must  decide  which 
among  many  possible  goals  it  wishes  to  pursue: 

a.  To  validate  ASVAB  scores  as  predictors  of  job  perform¬ 
ance  ("simple  validation  goaU: 

b.  To  determine  what  levels  of  ASVAB  predict  various 
levels  of  job  performance  (impact  of  standards  on  mission 
capability  goal); 

c.  To  determine  what  levels  of  ASVAB  predict  various 
levels  of  job  performance  (mission  capability)  at  specified 
costs  (cost-benefit  analysis  goal);  or 

d.  To  determine  allocation  of  personnel  across  AFSs  by 
optimizing  cost-benefit  ratios  and  utilization  of  talent 
(force  structure  cost-benefit  goal), 

2.  Secondly,  the  goal-stating  process  is  complicated  due  to  Air 
Force  institutional  policy  issues  which  are  entirely  separate 
from  the  methodological  issues  of  model  building.  More 
specifically,  although  the  interviews  with  policy  makers  that 
were  conducted  (in  the  course  of  developing  this  paper)  re¬ 
vealed  a  definite  need  for  a  mechanism  (such  as  a  linkage 
model)  to  evaluate  the  consequences  of  the  standard  setting 
process,  responses  during  interviews  firmly  established  a  low 
probability  of  use  (and  therefore  impact)  of  linkage  approaches 
in  the  near  term.  In  short,  the  payoff  of  this  AFHRL  work  will 
be  limited  until  participants  in  the  selection  and  classification 
system  are  sufficiently  convinced  of  its  merit  to  agree  to  adopt 
the  proposed  linkage  mechanism. 
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3.  Once  the  goal  or  set  of  goals  is  identified,  available  linkage 
models  can  be  evaluated  for  suitability  in  terms  of  quality, 
requirements,  and  capabilities.  For  the  four  models  reviewed 
in  this  paper,  the  value  of  the  model  results  generally  depend¬ 
ed  on  the  nature  of  the  job  performance  measures  that  were 
used  as  input  (i.e.,  norm-referenced  or  criterion-referenced 
scores).  Norm-referenced  job  performance  tests  can  be  used  in 
some  linkage  approaches,  but  for  some  goals  such  as  1(b)- 1(d) 
above,  results  are  not  easily  interpretable  since  the  quality  of 
job  performance  in  some  absolute  sense  is  unknown  (e.  g.,  how 
much  of  the  job  content  the  airman  has  mastered).  Some 
approaches  have  adopted  a  different  conception  of  the  quality 
of  job  performance  (i.  e.,  different  from  job  content  mastery) 
and  attempted  to  use  indicators  such  as  the  dollar  value  of 
performance  and  time  to  proficiency  in  lieu  of  criterion- 
referenced  performance  tests.  Those  approaches  have  been 
shown  to  have  inherent  limitations  of  a  different  type, 
however. 

By  way  of  a  top-level  concluding  summary.  Tables  3  through  5  are 
offered  to  evaluate  available  models  against  potential  goals,  using  the 
criteria  elaborated  earlier  in  this  paper.  Following  that,  recommendations 
for  future  work  in  this  area  are  proposed. 
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Table  3.  Evaluative  Comparison  of  Models  Assuming  Goal  1(b) 

To  determine  impact  of  enlistment  standards  on  job 
performance  (mission  capability) 


_ Model _ 

Exp  B-E  Rand 

Criteria  Chart  Anal^  TTP®  Models  Comments 


1.  Suitability  of 

Last  three  models 

approach  to  goal 

are  overmatched 
to  goal 

2.  Availability  of 

Competency-based 

needed  inputs  for 

tests  required  as 

Air  Force  appiica- 

* 

well  as  minimal 

tion 

competence  stand- 

ard 

3.  Quality  of 

Same  comments  as 

available  inputs 

* 

above 

4.  Processing  effi- 

ciency/flexibility 

*** 

5.  Usefulness  of 

No  cost 

results 

* 

information 

Note.  The  extent  to  which  models  meet  criteria  are  denoted  as: 


high 

medium 

low 


^Model  not  appropriate  for  goal. 


Table  4.  Evaluative  Comparison  of  Models  Assuming  Goal  1(c) 


To  determine  impact  of  enlistment  standards  on  mission 
capability  in  cost-benefit  terms 


Criteria 

Exp 

Chart® 

Model 

B-E 

^  Anal  TTP 

Rand 

Model 

Comments 

1.  Suitability 

of 

approach  to  goal 

*** 

*** 

2.  Availability  of 

SDy  a  problem  for 

needed  inputs  for 

B-E  Anal;  criterion- 

Air  Force 

applica- 

** 

referenced  measures 

tion 

a  problem  for  TTP 

3.  Quality  of 

Same  as  above 

available 

inputs 

♦ 

** 

** 

plus  supervisor 

bias  a  problem 

for  TTP 

4.  Processing 

I  efficiency/flexibility 

*** 

** 

5.  Usefulness 

of 

results 

** 

** 

*** 

Note. 

The  extent  to  which  models  meet  criteria  are  denoted  as: 

*** 

high 

** 

medium 

* 

low 

“Model 

not  appropriate 

for  goal. 
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Table  5.  Evaluative  Comparison  of  Models  Assuming  Goal  1(d) 


To  determine  impact  of  enlistment  standards  on  mission 
capability  in  cost-benefit  terms  across  AFSCs 

Meeting  this  goal  would  require  consequential  extensions  of  the 
three  approaches  that  involve  cost-benefit  analyses;  therefore, 
all  evaluation  results  have  a  high  degree  of  uncertainty  since 
the  feasibility  of  such  extensions  is  essentially  unknown.  As  a 
result,  indicators  (high,  medium,  low)  should  be  interpreted  as 
probability  estimates  rather  than  conclusive  assessments. 

_ Model _ 


Exp 

Criteria  Chart^ 

B-E 

Anal 

TTP 

Rand 

Model 

Comments 

1.  Suitability  of 
approach  to  goal 

* 

Rand  approach  has 
been  formulated 

2.  Availability  of 

needed  inputs  for  Air 

Force  application 

* 

** 

3.  Quality  of 

available  inputs 

* 

** 

** 

4.  Processing 

efficiency/flexibility 

** 

** 

** 

5.  Usefulness  of 
results 

** 

*** 

*** 

1 

Note.  The  extent  to  which  models  meet  the  criteria  are  denoted  as: 


high 

medium 

low 


^Model  not  appropriate  for  goal. 


Based  on  the  comparative  evaluations  presented  in  the  prior  two 
sections,  recommendations  for  future  work  are  as  follows: 

1.  AFHRL  should  determine  which  linkage  goal  or  set  of  goals  to 
pursue  based  on  criteria  such  as  the  following: 

a.  Is  the  goal  useful  for  the  Air  Force  at  large? 

b.  Is  the  work  involved  in  meeting  the  goal(s)  justified  in 
terms  of  "user  support"  from  the  manpower,  personnel 
and  training  (MPT)  community? 

c.  Is  the  likelihood  of  success  in  meeting  the  goal(s)  suffic¬ 
iently  high  to  warrant  a  sustained  effort? 

2.  Once  the  goal  state  is  specified,  AFHRL  should  determine  which 
linkage  models  are  most  applicable  (by  referring  initially  to 
tables  2  through  5)  and  proceed  to  examine  known  model 
weaknesses  to  assess  the  extent  of  the  weakness;  e.  g.,  test  the 
feasibility  of  using  continuous  (versus  dichotomous)  perform¬ 
ance  scores  in  the  Rand  approach  using  Air  Force  data.  In 
addition,  the  following  linkage  model  work  is  recommended: 

a.  Using  performance  data  from  the  eight  Air  Force  JSP 
specialties,  execute  the  models  for  linking  standards  to 
performance  and  conduct  comparative  analyses  of  the 
results, 

b.  Determine  procedures  for  assigning  a  dollar  value  to 
airman  performance  (as  the  benefit  in  a  cost-benefit 
ratio)  other  than  the  standard  deviation  of  salary  metric 
(SDy),  which  is  typically  used  in  the  general  utility 
equation. 

3.  AFHRL  should  also  explore  the  feasibility  of  deriving  ASVAB/ 
new  predictor-performance  test  relationships  for  specific  job 
content  areas.  Such  analyses  would  help  define  ASVAB's 
ability  to  predict  job  performance  and  also  help  develop  tests 
that  predict  performance. 


4.  Since  the  most  serious  limitation  for  most  models  concerns  the 
nature  of  the  job  performance  measurements  (i.  e.,  norm- 
referenced  or  criterion-referenced),  some  specific  recommend¬ 
ations  in  this  regard  include  the  following: 

a.  Explore  methods  for  scaling  performance  scores  to  define 
job  mastery,  especially  tO  define  what  constitutes 
minimally  acceptable  performance. 

b.  Extend  the  TTP  work  to  larger  samples  of  AFSs,  super¬ 
visors,  and  airmen  to  more  rigorously  examine  issues  of 
reliability  and  validity  and  the  general  defensibility  of 
time  to  proficiency  as  a  surrogate  job  performance  mea¬ 
sure.  In  addition,  scaling  TTP  performance  scores  to 
achieve  standardized  criterion-referenced  measures  is 
required  to  have  confidence  that  these  data  are  useful  for 
setting  enlistment  standards. 
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APPENDIX  A:  PROCEDURES  FOLLOWED  IN  DEVELOPING  TfflS  PAPER 


Relevant  literature  for  this  paper  was  identified  through  searches  of 
the  Defense  Technical  Information  Center  (DTIC),  the  published  and 
unpublished  reports  of  the  Joint-Service  Job  Performance  Measurement/ 
Enlistment  Standards  Project  (JSP),  and  the  available  reports  on  Air  Force 
Person-Job  Match  research  efforts.  The  JSP  literature  includes  papers 
prepared  for  the  Committee  on  the  Performance  of  Military  Personnel/ 
National  Research  Council/National  Academy  of  Sciences  and  all  relevant 
RAND  and  Center  for  Naval  Analyses  (CNA)  publications.  Also  included  in 
the  review  are  personal  communications  from  JSP  Working  Group  Meetings 
and  similar  occasions  as  veil  as  relevant  publications  from  the  profes¬ 
sional  literature  in  the  fields  of  psychometrics  and  personnel/industrial/ 
organizational  psychology. 

Sources  from  the  literature  were  supplemented  and  verified  by 
structured  interviews  with  key  Air  Force  personnel  involved  in  the  busi¬ 
ness  of  recruiting,  selection,  classification,  and  training.  Interviews  were 
based  on  a  questionnaire  that  covered  the  following  topics; 

1.  The  enlistment/classification  process  as  it  works  today; 

2.  The  desired  characteristics  of  a  candidate  linkage  methodology 
that  could  improve  that  process; 

3.  The  operational  constraints  that  would  affect  the  implementa¬ 
tion  of  such  a  methodology; 

4.  Meaningful  evaluation  criteria  for  linking  methodologies; 

5.  Completeness  of  sources  identified  for  the  literature  review; 
and 

6.  Completeness  of  proposed  offices/individuals  for  the  structured 
interviews. 

Offices  where  interviews  were  conducted  are  listed  below  and  a  copy 
of  the  questionnaire  used  as  a  basis  for  the  interview  follows. 

Using  the  questionnaires  described  in  the  previous  paragraph,  struc¬ 
tured  interviews  were  conducted  with  various  Air  Force  offices  involved 
with  the  initial  selection  and  classification  process.  Among  those  inter¬ 
viewed  included  key  personnel  at  the  following  offices: 
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Air  Force  Recruiting  Service  (APRS)  working  in  the  develop¬ 
ment  and  application  of  the  Procurement  Management 
Information  System  (PROMTS)  and  the  Processing  and 
Classification  of  Enlistees  (PACE)  system  (Mr.  Joe  Manuel  - 
AFRS/RSR); 

Air  Force  Military  Personnel  Center  (AFMPC)  working  with  job 
classification  and  assignments  (Lt  Col  Dave  Richmond,  Jr.  — 
AFMPCAi)PMRPQ;  Capt  Mike  Begley  —  AFMPC/DPMRPQ;  Capt 
Wendy  Campo  —  AFMPC/DPMRPQ;  Capt  Doug  Eads  —  AFMPC/ 
DPMRPQ;  Maj  Russ  Seeman  --  AFMPC/DPMYFX;  Lt  Col  Rick 
Creekmore  -  AFMPC/DPMRA;  Capt  Larry  Letcher  —  AFMPC/ 
DPMRA); 

Headquarters  United  States  Air  Force,  Deputy  Chief  of  Staff  for 
Personnel,  Directorate  of  Personnel  Plans,  Policy  Division  (HQ 
USAF/DPXOA)  working  with  accession,  selection,  and  classi¬ 
fication  policy,  and  the  Air  Force  job  performance  measure¬ 
ment  program  (Lt  Col  Bill  Strickland  -  USAF/DPXOA;  Lt  Col 
Doug  Gorman  -  USAF/DPXOA); 

Air  Training  Command  (ATC)  working  with  the  PROMIS,  PACE, 
training  and  policy-making  process  (Col  Fraine  Zeitler  -  ATC/ 
TTO;  Lt  Col  Ed  Cecconi  --  ATC/TTPR;  Maj  Bill  Cummings  -  ATC/ 
XPRR;  Capt  Bernard  Aziu  —  ATC/XPRR); 

Office  of  the  Deputy  Assistant  Secretary  of  Defense  for  Military 
Manpower  and  Personnel  Policy  (DASD/MM&PP)  working  with 
accession  policy  and  joint-service  job  performance  measure¬ 
ment  (Lt  Col  Dickie  Harris  —  OASD/FM&P);  and 

AFHRL  working  with  job  performance  measurement  research 
and  the  selection,  classification,  and  retraining  process  (Lt  Col 
Dan  Leighton  -  AFHRL/MO;  Lt  Col  Nick  Ovalle  -  AFHRL/IDE; 

Lt  Col  Rodger  Ballentine  -  AFHRL/ID;  Capt  Marty  Pellum  - 
AFHRL/IDE;  Capt  Eric  Duncan  —  AFHRL/IDE;  Dr.  Manuel  Pina  — 
AFHRL/MOMD;  Mr.  Larry  Looper  -  AFHRL/MOMD;  Dr.  Henk 
Ruck  -  AFHRL/ID;  Dr.  Joe  Weeks  -  AFHRL/PR;  Dr.  Bill  Alley 
-  AFHRL/MOT;  Dr.  Sherrie  Gott  -  AFHRL/MOMJ). 


QUESTIONNAIRE 


for 


Exploratory  Models  to  Link  Job  Performance 
to  Enlistment  Standards 


Contract:  F41689-86-D-0052 
Task:  0  9 


Prepared  for:  Air  Force  Human  Resources  Laboratory 
AFHRL/IDE 
Captain  Eric  Duncan 
August  17,  1987 
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Privacy  Act  Statement:  You  have  been  selected  to  participate  m  an 
Air  Force  interview.  Under  the  Privacy  Act,  your  participation  is 
completely  voluntary  and  no  adverse  action  can  be  taken  against 
anyone  who  chooses  not  to  participate.  The  results  of  these  inter¬ 
views  will  be  converted  to  statistical  data  and  no  individual 
responses  will  ever  be  identified.  Your  responses  are  completely 
confidential. 


Introduction 


The  purpose  of  this  questionnaire  is  to  collect  information  needed  for 
the  following:  (a)  to  better  understand  the  Air  Force  selection/classification 
process  and  the  inherent  organizational,  operational,  and  policy  limitations, 
(b)  to  develop  meaningful  criteria  to  evaluate  existing  technology  and  can¬ 
didate  models  dealing  with  job  performance  measurement  and  accession 
attributes,  and  (c)  to  determine  the  feasibility  of  using  such  technology 
and  models  within  the  Air  Force  context. 

Please  fill  in  the  following  information  and  then  read  and  answer  the 
questions  in  the  remaining  sections. 

Date:  _ 

Location: _ 


Time: _ 

Name  and  Office  Symbol  of  Person(s)  Responding  to  the 
Questionnaire: 


Name  of  Person  Conducting  the  Interview: 
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QUESTIONS 


1.  Brian  K.  Waters,  Janice  H.  Laurence,  and  Wayne  J.  Camara  recently 
prepared  a  paper  entitled  "Personnel  Enlistment  and  Classification 
Procedures  in  the  U.  S.  Military"  for  the  Committee  on  the  Performance  of 
Military  Personnel,  Commission  on  Behavioral  and  Social  Sciences  and 
Education,  National  Research  Council  (National  Academy  Press,  Washington, 
D.  C.,  1987).  In  this  paper,  they  describe  the  Air  Force  structure  for 
accession  policy  management  with  the  following  description  and  figure; 

Figure  4  shows  the  Air  Force  system.  The  Air  Force  policy 
management  level  is  the  Accession  and  Reenlistment  Policy  Branch 
(AF/DPXOA)  located  in  the  Pentagon.  As  its  name  suggests,  policy 
analysts  in  this  office  serve  as  the  staff  advisors  to  the  Air  Force 
Deputy  Chief  of  Staff  for  Personnel  on  accession  policy  matters.  The 
setting  of  standards  for  selection  takes  place  in  this  office.  The 
Accessions  and  Reenlistment  Policy  Branch  (AF/DPXOA)  coordinates 
closely  with  other  Air  Force  headquarters  staff  agencies  and  DOD 
accession  policy  personnel.  The  Air  Force  OPR  for  classification 
standards  is  the  Classification  and  Training  Branch  (AFMPC/ 

DPMRTC)  at  the  Air  Force  Military  Personnel  Center . Like  the 

Air  Force,  the  responsibility  for  setting  Army  standards  is  delegated 
to  that  Service,  yet  is  monitored  by  OASD/FM&P/AP. 
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Figure  4.  Air  Force  Selection  and  Classification  Standards  Management. 


66 


Is  this  structure  accurate?  Yes  _  No  _  If  no,  what  changes  are 

recommended?  (Make  changes  directly  to  Figure  4  if  appropriate  or  list 
proposed  changes  in  the  following  space); 


Is  the  description  accurate?  Yes  _  No  _  If  no,  what  changes 

are  recommended?  (Make  changes  directly  to  the  description  or  list 
proposed  changes  in  the  following  space): 


2.  During  the  "Literature  Review"  phase  of  this  effort,  relevant 
documents  were  identified  using  the  Defense  Technical  Information  Center 
(DTIC),  unpublished  materials  within  the  Department  of  Defense's  Job 
Performance  Measurement  Project,  publications  from  Rand  and  the 
National  Academy  of  Sciences,  and  private  literature  sources.  Approxi¬ 
mately  25  of  these  documents  were  collected  and  are  listed  in  Attachment 
1.  Please  scan  these  references  to  evaluate  the  completeness  of  the 
literature  search.  Are  there  any  other  references  which  should  be 

included?  Yes  _  No  _  If  yes,  what  additional  references  are 

recommended? 


3.  For  each  of  the  major  candidate  models  identified  within  the 
references,  plans  are  to  compile  the  following  information: 

a.  Type  of  model 

b.  Summary  and  description  of  model/algorithm 

c.  Assumptions  for  use 

d.  Operational  constraints  and  limitations 

e.  Inputs  required 

f.  Outputs  provided 

g.  Air  Force  data  required:  Data  available?  Continue  to  be 
available?  Will  data  be  available  in  the  future?  Data  location? 

h.  Model  acceptable  to  Air  Force? 

i.  Feasibility  of  being  used  by  Air  Force  policy  makers. 

j.  How  is  job  performance  measured? 

k.  What  is  the  linkage  between  job  performance  and  accession 
attributes? 

l.  Are  there  objective  criteria  to  evaluate  alternative  policies? 

m.  How  to  validate  model  in  order  to  demonstrate  its  applicability 
and  reliability  to  Air  Force  decision  makers. 

Are  there  other  areas  that  should  be  investigated  and/or  included? 
Yes  _  No _  If  yes,  what  are  those  areas? 


Are  you  aware  of  any  models  or  analytical  techniques  currently 
being  used  to  set  selection/classification  standards? 


In  your  opinion,  what  are  the  operational  constraints  and  other 
factors  (such  as  policy,  technical,  or  practical  issues)  which  must  be 
considered  when  setting  selection/classification  standards? 


In  your  opinion,  in  order  to  evaluate  candidate  models  what  are  the 
most  meaningful  criteria  (e.  g.,  cost  per  productive  man-month,  productive 
capacity  of  the  force,  and  minimum  accession  quantity  and  quality  levels) 
to  consider? 


How  would  you  prioritize  these  criteria? 


Do  you  have  any  ideas  or  recommendations  concerning  how  the  Air 
Force  should  go  about  linking  job  performance  measurement  and  enlisted 
standards? 


In  general,  do  you  believe  there  are  any  other  aspects  of  this  effort 
that  we  ought  to  be  considering? 


Any  general  comments  to  improve  this  effort? 


4.  This  questionnaire  will  be  administered  to  personnel  at  the;  Air 
Force  Human  Resources  Laboratory  (AFHRL)  involved  in  job  performance 
measurement  research  and  the  selection/classification  process;  Air  Force 
Military  Personnel  Center  (AFMPC)  involved  in  the  determination  of  job 
classification;  Air  Force  Recruiting  Service  (APRS)  involved  in  the  develop¬ 
ment  and  application  of  the  Procurement  Management  Information  System 
(PROMIS)  and  the  Processing  and  Classification  of  Enlistees  (PACE)  system; 
Air  Training  Command  (ATC)  involved  with  the  PROMIS,  PACE,  training 
and  policy-making  processes;  Headquarters  USAF,  DCS/Personnel,  Director¬ 
ate  of  Personnel  Plans,  Policy  Division  (AFA)PXOA)  involved  in  the  acces¬ 
sion,  selection  and  classification  policy  and  the  Air  Force  job  performance 
measurement  program;  and  the  Office  of  the  Assistant  Secretary  of 
Defense,  Force  Management  and  Personnel  (OASD/FM&P)  involved  in 
accession  policy  and  job  performance  measurement.  Specific  individuals  to 
be  contacted  include  the  following: 

a.  AFHRL  -  Lt  Col  Dan  Leighton,  Dr.  Manuel  Pina,  Mr. 

Larry  Looper,  Capt  Mark  Emerson,  Lt  Col  Nick 

Ovalle,  Capt  Marty  Pellum,  Dr.  Henk  Ruck,  Dr. 

Joe  Weeks,  Dr.  Bill  Alley,  Lt  Col  Rodger 

Ballentine 

b.  ATC-  Col  Fraine  Zeitler,  Lt  Col  Ed  Cecconi,  Maj  Bill 

Cummins 

c.  AFMPC  -  Lt  Col  D.  W.  Richmond,  Jr. 

d.  AFRS  -  Lt  Col  Frank  Terrell 

e.  AF/DPXOA  -  Lt  Col  Doug  Gorman 

f.  OASD/FM&P  -  Lt  Col  D.  A.  Harris 
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APPENDIX  C:  ANNOTATED  BIBLIOGRAPHY 


ANNOTATED  BBLIOGRAPHY 


Alley,  W.  E.  (1987).  Ocrupational  learning  difficulty.  Paper  prepared  for 
the  Committee  on  the  Pei^ormance  of  Military  Personnel,  Com.;;- ;sion 
on  Behavioral  and  Social  Sciences  and  Education,  National  Research 
Council,  Washington,  D.  C. 

The  present  Joint-Service  Job  Performance  Measurement/ 
Enlistment  Standards  Project  has  included  only  a  few  out  of  the 
many  military  entrance  level  occupations.  Until  such  time  that 
sufficient  data  concerning  the  linkages  between  aptitude  tests  and 
job  performance  measures  have  been  collected  to  cover  all  or  most 
specialties,  the  author  describes  "a  conceptual  framework  and  data 
analysis  approach"  to  extend  the  utilization  of  existing  data  to  include 
additional  specialties.  The  theory  is  taken  from  the  Air  Force 
Learning  Difficulty  Project.  The  concepts  of  job  learning  difficulty, 
benchmark  learning  difficulty  scales,  and  average  task  difficulty  per 
unit  time  spent  (ATDPUTS)  are  presented  to  provide  a  way  to 
compare  specialties  with  regard  to  a  common  metric.  The  Air  Force 
Learning  Difficulty  Project  "can  serve  as  a  timely  and  cost-effective 
model  for  extending  the  results  of  this  work  to  specialties  that  have 
not  been  included  in  the  empirical  field  studies." 

Allred,  L.  J.  (1986).  Alternatives  to  the  validity  coefficient  for  reporting 
the  test-criterion  relationship.  Paper  prepared  for  the  Committee  on 
the  Performance  of  Military  Personnel,  Commission  on  Behavioral 
and  Social  Sciences  and  Education,  National  Research  Council, 
Washington,  D.  C. 

This  paper  describes  various  methods  for  displaying  detailed 
information  about  the  relationship  between  test  scores  and  job 
performance  measures.  Several  sets  of  hypothetical  data  are  used  to 
illustrate  methods  ranging  from  a  full  plot  of  test  and  performance 
scores  to  grouping  test  scores  into  intervals  and  then  displaying 
distributions  of  test  scores  within  those  intervals.  Expectancy 
methods  (chart,  table,  and  plot),  used  in  the  evaluation  of  the 
prediction  of  two-level  criteria  (e.  g.,  success  versus  failure),  and 
frequency  tables,  used  in  determination  of  maximum  correct 
classifications,  are  also  presented.  The  author  concludes  by  stating 
"these  methods  are  easily  extended  to  any  test-criterion  situation. 


The  final  choice  of  display  methods  should  depend  upon  both  test 
user  needs  and  level  of  psychometric  expertise." 


Armor,  D.  J.  (1985,  October).  Linking  job  performance  to  enlistment 

standards  Continuous  vs.  pass-fail  models,  i-aper  ^icsented  to  the 
Joint-Service  '' -irking  Group. 

In  using  models  to  link  job  performance  with  enlistment 
standards,  this  paper  deals  with  the  issue  of  using  performance 
scores  that  are  transformed  into  some  type  of  categorical  measure, 
such  as  a  pass-fail  dichotomy  based  on  a  minimum  acceptable 
performance  standard,  versus  a  continuous  score  approach.  The 
Rand  enlistment  standards  model  is  used  to  illustrate  the  use  of 
continuous  scores  and  to  compare  results  with  the  pass-fail  or 
minimum  standard  approach.  Preliminary  comparisons  using  Army 
data  show  the  continuous  scoring  approach  gives  results  close  to 
those  using  the  pass-fail  approach. 

Armor,  D.  J.  (1987,  March).  Job  performance  measurement  and  defense 
manpower  management.  In  H.  G.  Baker  &  G.  J.  Laabs  (Eds.), 
Proceedings  of  the  Department  of  Defense/Educational  Testing 
Service  Conference  on  Job  Performance  Measurement  Technologies. 
San  Diego,  CA:  Navy  Personnei  Research  and  Development  Center. 

This  report  reflects  comments  made  by  the  author  as  the 
keynote  speaker  during  the  opening  ceremonies  of  the  Job 
Performance  Measurement  Technologies  Conference  held  in  March, 
1987,  in  San  Diego,  California.  Serving  as  the  Principal  Deputy 
Assistant  Secretary  of  Defense,  Force  Management  and  Personnel,  Dr. 
Armor  spoke  of  four  policy  areas  where  the  application  of  job 
performance  me'’surement  can  have  major  benefits  for  manpower 
management. 

The  first  area  deals  with  accession  policy  and  enlistment 
standards;  i.  e.,  validating  enlistment  standar.'  ,  which  is  the  original 
goal  of  the  Joint-Service  Project.  The  second  area,  also  a  validation 
effort,  focuses  on  the  relationship  between  training  time  and  unit 
proficiency,  based  on  actual  unit  performance.  Evidence  of  a  strong 
relationship  here  would  provide  better  justification  to  Congress  for 
funding  requirements  for  various  training  programs.  The  third  area 
centers  upon  readiness  and  force  mix  decisions  and  the  need  to 
improve  unit  readiness  assessment;  i.  e.,  "to  move  from  ratings  of 
resources  to  measures  of  actual  individual  and  unit  performance." 


^4 


The  fourth  area  involves  the  acquisition  of  weapon  systems.  In  this 
case,  the  need  is  to  evaluate  human  performance  as  well  as  machine 
performance  in  considering  total  weapon  system  performance.  The 
potential  payoffs  for  the  Joint-Service  Project  include  "improving 
manpower  management  practice,  resolving  policy  disputes,  and 
defending  our  programs  to  Congress," 

Armor,  D.  J.,  Fernandez,  R.  L„  Bers,  K.,  &  Schwarzbach,  D.  (1982).  Recruit 
aptitudes  and  Army  job  performance  (R-2874-MRAL),  Santa  Monica, 
CA;  Rand  Corporation, 

After  incorrect  scoring  procedures  for  the  Armed  Services 
Vocational  Aptitude  Battery  (ASVAB)  between  1976  and  1980  were 
identified,  this  Rand  study  was  initiated  to  investigate  the  relation¬ 
ship  between  recruit  aptitudes  and  job  performance  in  Army  jobs. 

The  key  policy  question  was  whether  revised  enlisted  standards 
would  provide  for  a  cost-effective  manpower  force.  To  address  this 
question,  a  cost-effective  tradeoff  model  was  developed  and  tested 
using  data  for  Army  Infantrymen.  The  model  is  built  upon  the 
concept  of  a  qualified  man-month,  defined  as  the  number  of  months 
contributed  by  individuals  who  complete  training,  remain  in  the 
Army,  and  can  pass  an  on-the-job  Skill  Qualification  Test.  By 
considering  various  costs  for  recruiting,  training,  and  maintaining 
personnel,  the  model  is  used  to  compare  various  ability  mixes  to 
determine  the  most  cost-effective  enlistment  standard.  Detailed 
formulas  and  sample  computer  output  are  provided. 

Burtch,  L.  D.,  Lipscomb,  M.  S.,  &  Wissman,  D.  J.  (1982).  Aptitude 

requirements  based  on  task  difficulty:  Methodology  for  evaluation 
(AFHRL-TP-81-34,  AD-AllO  568).  Brooks  AFB,  TX:  Manpower  and 
Personnel  Division,  Air  Force  Human  Resources  Laboratory. 


The  development  and  application  of  a  technology  designed  for 
the  evaluation  of  the  difficulty  of  Air  Force  jobs  in  conjunction  with 
the  aptitude  level  required  for  the  job  are  described.  The  technology 
developed  makes  use  of  computed  variables  and  task  factor  data 
collected  by  the  Air  Force  Occupational  Measurement  Center  as  well 
as  benchmark  difficulty  data  collected  by  contract  personnel  experts 
for  the  specialties  under  study.  The  application  of  this  technology 
provides  a  unique  method  of  determining  and  comparing  the 
learning  difficulty  of  Air  Force  tasks  and  jobs,  both  within  and  across 
career  specialties.  Analyses  have  indicated  high  interrater  relia¬ 
bilities  for  both  supervisory  and  benchmark  ratings.  A  two-variable 


multiple  regression  equation  was  developed  for  each  of  the  spec¬ 
ialties  studied.  Relatively  high  correlations  were  obtained  between 
the  two  ratings,  indicating  that  independent  raters  tend  to  agree  with 
supervisors.  These  equations  resulted  in  estimates  of  average  task 
difficulty  per  unit  time  (ATDPUT)  values  for  each  job  in  each  spec¬ 
ialty.  The  value  of  these  estimates  and  implications  for  their  use  are 
discussed  (authors'  abstract). 

Carpenter,  M.  A.,  &  Monaco,  S.  J.  (1987).  Time  to  proficiency.  Paper 
prepared  for  the  Training  Systems  Division,  Air  Force  Human 
Resources  Laboratory,  Brooks  Air  Force  Base,  Texas. 

The  Time  to  Proficiency  (TTP)  project  has  two  major  objectives: 
"(1)  to  determine  the  feasibility  and  validity  of  using  supervisor 
estimates  as  measures  of  job  performance  and  to  estimate  the  extent 
to  which  mental  aptitudes  affect  job  performance;  and  (2)  to  develop 
and  apply  a  prototype  analytical  model  to  evaluate  how  changing 
aptitude  requirements  based  on  job  performance  information  would 
affect  occupational  capability;  manpower,  personnel  and  training 
(MPT)  policies;  MPT  programs;  and  MPT  costs."  Using  data  collected 
from  supervisors  within  one  Air  Force  specialty  (Avionics 
Communications  Specialist),  regression  techniques  were  used  to 
estimate  the  effects  of  aptitude  on  productivity,  while  controlling  for 
experience.  A  prototype  model  was  then  developed  by  integrating 
the  functional  and  logical  relationships  of  four  component  submodels 
(production,  attrition,  cost,  and  distribution  of  aptitude).  Application 
of  the  prototype  model  yielded  an  optimum  level  of  aptitude  which 
minimizes  cost  per  unit  of  production.  Results  appear  promising; 
however,  further  refinement  and  testing  are  recommended. 

Cascio,  W.  F.  (1987).  Alternative  uses  of  performance  data  in  manpower 
management.  Paper  prepared  for  the  Committee  on  the  Performance 
of  Military  Personnel,  Commission  on  Behavioral  and  Social  Sciences 
and  Education,  National  Research  Council,  Washington,  D.  C. 


\  variety  of  different  models  using  job  performance  measures 
are  presented  in  this  paper  including  an  annotated  bibliography  on 
manpower  management  and  human  resource  planning.  From  the 
literature  review,  the  author  concludes  that  "  many  authors  talk 
about  the  importance  of  the  relationship  between  expected  job 
performance  and  manpower  forecasts,  yet  no  one  has  offered  any 
method  of  doing  so.  "  and  possible  methods  are  then  suggested. 


Considerable  attention  is  given  to  the  alternative  metrics  of  job 
performance  and  a  conclusion  that  "  a  measure  of  competency  or 
level  of  job  mastery  is  essential,  "  Four  different  ways  of  expressing 
utility  (the  overall  payoff  associated  with  the  use  of  a  selection 
procedure)  are  discussed:  dollars,  percentage  increases  in  output, 
expected  reductions  in  the  number  of  hires  required  to  do  a  job,  and 
the  labor  cost  savings  associated  with  the  reduced  hires. 

Two  methods  are  presented  to  relate  job  performance  to 
enlistment  standards:  expectancy  charts  and  break-even  analysis. 

The  author  concludes  that  "  the  technology  for  relating  performance 
data  to  enlistment  standards  does  exist,  although  some  variations 
and  extensions  of  traditional  methodology  are  necessary  in  the 
context  of  military  jobs. 

Department  of  the  Air  Force  (1985).  Quality  requirements  for  Air  Force 
military  enlisted  personnel.  Paper  prepared  for  the  Office  of  the 
Assistant  Secretary  of  Defense,  Force  Management  and  Personnel, 
Washington,  D.  C. 

The  1985  Omnibus  Defense  Authorization  Act  directs  the 
Secretary  of  Defense  to  submit  to  the  House  and  Senate  Committees 
on  Armed  Services  a  report  detailing  the  current  quality  of  the 
enlisted  force  in  each  Service,  the  projected  quality  requirements  for 
enlisted  personnel  in  each  Service  over  the  next  5  years,  the 
recruiting  and  retention  levels  necessary  over  the  next  5  years  to 
sustain  those  quality  requirements  in  each  Service,  and  any  changes 
to  current  compensation  policies  necessary  to  sustain  those  quality 
requirements.  This  report  provides  the  Air  Force  position  on  each  of 
these  areas  (author's  abstract). 

Dickinson,  T.  L.,  Hedge,  J.  W.,  &  Ballentine,  R.  D.  (1987,  March). 

Predictability  of  the  ASVAB  for  the  Air  Force's  job  performance 
measurement  system.  In  H.  G.  Baker  &  G.  J.  Laabs  (Eds.),  Proceedings 
of  the  Department  of  Defense/  Educational  Testing  Service 
Conference  on  Job  Performance  Measurement  Technologies.  San 
Diego,  CA:  Navy  Personnel  Research  and  Development  Center. 

The  Air  Force  Human  Resources  Laboratory  (AFHRL)  is 
currently  developing  a  measurement  system  for  obtaining  job 
performance  data.  This  job  performance  measurement  system 
(JPMS)  will  serve  three  interrelated  purposes.  First,  the  JPMS  will 
provide  operational  managers  of  the  Air  Force's  human  resources 
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program  with  criteria  to  evaluate  program  effectiveness.  Second, 
the  JPMS  will  provide  Air  Force  research  scientists  with  performance 
measures  to  use  in  research  &  development  (R&D)  projects.  Finally, 
the  JPMS  will  provide  measures  for  assessing  the  predictability  of 
the  Armed  Services  Vocational  Aptitude  Battery  (ASVAB)  for  on-the- 
job  performance.  This  paper  examined  the  underlying  structure  of 
the  JPMS,  and  the  predictability  of  the  ASVAB  for  the  JPMS  (authors’ 
abstract). 

Fernandez,  R.  L.  (1985).  Setting  enlistment  standards  and  matching 

recruits  to  jobs  using  iob  performance  criteria  (R-3067-MIL).  Santa 
Monica,  CA:  Rand  Corporation. 

This  report  describes  the  research  efforts  of  a  two-year  study 
carried  out  by  the  Rand  Corporation’s  Defense  Manpower  Research 
Center  for  the  Office  of  the  Secretary  of  Defense.  The  results  build 
upon  the  earlier  work  of  Armor  et  al.  (1982)  and  focus  on  the 
feasibility  of  the  Rand  cost/performance  tradeoff  model  for  the 
setting  of  enlistment  standards,  and  on  job  performance  measures  in 
matching  recruits  to  military  specialties. 

Considerable  detail  is  presented  in  examining  the  relationship 
between  individual  entry  characteristics  and  two  measures  of  job 
performance  -  attrition  and  the  Army's  Skill  Qualification  Test  (SQT). 
These  two  performance  measures  are  combined  into  one  single 
measure,  qualified  man-months,  which  is  the  key  concept  of  the 
Rand  model.  Results  using  four  Army  military  occupational 
specialties  (Infantryman,  Multichannel  Communications  Equipment 
Operator,  Tactical  Wire  Operations  Specialist,  and  Medical  Specialist) 
are  presented  and  analyzed. 

Con  tusions  are  that  the  model  does  serve  as  a  viable  tool  for 
the  setting  of  job  standards,  but  concerns  exist  with  the  "proportion¬ 
ality  assumption,"  the  choice  of  test  difficulty,  and  the  setting  of 
standards  objectively. 

Garcia,  S.  K.,  Ruck,  H.  W.,  &  Weeks,  J.  (1985).  Benchmark  learning 

difficulty  technology:  Feasibility  of  operational  implementation 
(AFHRL-TP-85-33,  AD-A161  797).  Brooks  AFB,  TX:  Manpower  and 
Personnel  Division,  Air  Force  Human  Resources  Laboratory. 

In  1973,  the  Air  Force  Manpower  and  Personnel  Center 
requested  that  the  Air  Force  Human  Resources  Laboratory  develop 
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an  objective  procedure  for  establishing  the  relative  aptitude 
requirements  for  enlistee  occupations.  After  10  years  of  extensive 
research,  the  Laboratory  developed  a  state-of-the-art  technology  for 
this  purpose.  The  technology  produces  measures  of  occupational 
learning  difficulty  that  can  be  used  as  a  job-centered  frame  of 
reference  for  establishing  aptitude  requirements.  These  measures 
were  used  as  an  empirical  basis  for  establishing  aptitude  require¬ 
ments  in  the  April  1982  revision  of  Air  Force  Regulation  39-1, 

Airman  Classification  Regulation.  Further,  occupational  learning 
difficulty  is  applied  within  Air  Force  person-job  match  algorithms  for 
determining  enlistee  job  assignments.  Because  occupational  learning 
difficulty  will  be  used  for  these  purposes  in  the  future,  it  is  impor¬ 
tant  that  the  associated  measurement  procedure  be  transferred  to  an 
operational  organization  for  routine  implementation.  To  transfer  the 
technology  it  was  first  necessary  to  investigate  the  feasibility  of 
having  Air  Force  personnel,  as  opposed  to  research  personnel,  collect 
benchmark  ratings  for  use  in  deriving  the  learning  difficulty  mea¬ 
sures.  For  this  purpose,  eight  members  of  the  Air  Force  Occupational 
Measurement  Center  (OMC)  collected  benchmark  ratings  for  nine 
enlisted  specialties  that  had  been  previously  evaluated  by  research 
personnel.  For  each  specialty,  a  measure  of  occupational  learning 
difficulty  was  derived.  Ratings  produced  by  the  OMC  team  were  then 
compared  to  ratings  collected  by  the  research  personnel  to  assess  the 
reliability  and  validity.  Results  of  the  analyses  indicated  that  the 
reliability  and  validity  of  the  OMC  ratings  were  equivalent  to  the 
reliability  and  validity  of  the  ratings  previously  collected  by  the 
research  personnel.  Hence,  this  study  supports  the  feasibility  of 
having  OMC  personnel  routinely  collect  benchmark  ratings  of 
learning  difficulty  (authors'  abstract). 

Gott,  S.  P.,  Lesgold,  A.,  &  Glaser,  R.  (1987,  March).  Implications  of 

cognitive  psychology  for  measuring  job  performance.  In  H.  G.  Baker  & 
G.  J.  Laabs  (Eds.),  Proceedings  of  the  Department  of  Defense/ 
Educational  Testing  Service  Conference  on  Job  Performance 
Measurement  Technologies.  San  Diego,  CA:  Navy  Personnel  Research 
and  Development  Center. 


In  comparison  to  a  well-developed  technology  for  aptitude 
measurement  and  selection  testing,  the  measurement  of  learned 
occupational  proficiency  is  underdeveloped.  The  problem  is 
especially  severe  for  the  Armed  Services  because  of  the  many  highly 
technical  jobs  involved  and  the  short  periods  of  enlistment  in  which 
both  training  and  useful  performance  must  take  place.  To  increase 


the  effectiveness  of  both  formal  training  and  on-the-job  learning,  we 
need  forms  of  assessment  that  provide  clear  indicators  of  the  content 
and  reliability  of  new  knowledge.  Since  many  of  the  military's  jobs 
have  a  major  cognitive  component,  the  needed  measurement 
methodology  must  be  able  to  deal  with  cognitive  skills.  In  this  paper 
we  assert  that  the  measurement  of  job  performance  should  be  driven 
by  modem  cognitive  theory  that  conceives  of  learning  as  the  acqui¬ 
sition  of  structures  of  integrated  conceptual  and  procedural  know¬ 
ledge.  At  various  stages  of  the  acquisition  process,  there  exist 
different  integrations  of  knowledge,  different  degrees  of  procedural 
skill,  differences  in  rapid  memory  access,  and  differences  in  the 
mental  representations  of  tasks  to  be  performed.  To  be  maximally 
useful,  proficiencey  measurement  must  be  based  upon  the  assess¬ 
ment  of  these  knowledge  structures,  information  processing  pro¬ 
cedures,  and  mental  representations.  Advancing  expertise  or 
possible  impasses  in  the  course  of  learning  will  be  signaled  by 
cognitive  differences  of  these  types. 

We  present  a  cognitive  account  of  the  components  of  skilled 
performance,  discuss  specific  measurement  procedures  that  have 
been  employed  in  various  domains  including  Air  Force  technical 
occupations,  and  illustrate  the  utility  of  the  measures  in  a  successful 
training  study  with  Air  Force  electronics  technicians  (authors' 
abstract). 

Green,  B.,  &  Shavelson,  R.  (1987,  March).  Distinguished  panel  discussion 
on  issues  in  the  joint-service  JPM  program.  In  H.  G.  Baker  &  G.  J. 

Laabs  (Eds.),  Proceedings  of  the  Department  of  Defense/Educational 
Testine  Service  Conference  on  Job  Performance  Measurement 
Technologies.  San  Diego,  CA:  Navy  Personnel  Research  and 
Development  Center. 

This  report  documents  the  comments  made  by  the  authors 
dunng  a  "distinguished"  panel  discussion  presented  on  the  third  and 
last  day  of  the  Job  Performance  Measurement  Technologies 
Conference  held  in  March,  1987,  in  San  Diego,  California.  Both 
authors  serve  on  the  Committee  on  the  Performance  of  Military 
Personnel,  National  Research  Council  of  the  National  Academy  of 
Sciences.  The  major  criticism  to  date  of  the  Joint-Service  Project  by 
the  Committee  has  been  the  concentration  on  the  "norm-referenced" 
approach  to  test  development  and  scoring;  criterion-referenced 
measures  are  recommended.  After  discussing  the  design  of  job 
pertormance  measurements,  the  authors  present  various  inter- 
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pretations  of  the  meaning  of  "linking  enlistment  standards  to  job 
performance."  Interpretations  range  from  a  "giant  validity  study"  to 
a  much  "harder  problem  of  comparing  the  worths  of  performances  on 
various  jobs."  For  the  more  difficult  interpretations  the  project  is  at 
"the  edge  of  current  scientific  methodology." 

Greer,  O.  L.,  &  Cascio,  W,  F.  (1987).  Is  cost  accounting  the  answer? 

Comparison  of  two  behaviorally  based  methods  for  estimating  the 
standard  deviation  of  job  performance  in  dollars  with  a  cost¬ 
accounting-based  approach.  Journal  of  Applied  Psychology,  72^4). 
588-595. 


Accurate  estimation  of  the  standard  deviation  of  job 
performance  in  dollars  (SDy)  can  improve  the  precision  of  utility 
estimates  of  expected  payoffs  from  personnel  programs.  The 
purpose  of  this  study  was  to  compare  directly  the  estimates  of  SDy 
obtained  using  a  cost-accounting-based  estimate  of  SDy,  the  Global 
Estimation  Model,  and  the  CREPID  procedure.  The  study  was 
conducted  in  a  large,  soft-drink  bottling  company.  Each  method  for 
estimating  SDy  was  applied  to  the  job  classification,  route  salesman, 
producing  three  independent  estimates  of  SDy.  These  estimates  were 
tested  for  significant  differences.  Results  indicated  that  the  Global 
Estimation  Model  estimate  and  the  cost-accounting-based  estimate 
were  not  significantly  different,  whereas  the  estimate  produced  by 
the  CREPID  procedure  was  significantly  smaller.  Limitations  of  the 
cost-accounting-based  estimate  are  identified  and  results  are 
discussed  in  terms  of  their  implications  for  the  theory  and  practice  of 
utility  analysis  in  organizations  (authors’  abstract). 

Harris,  D.  A.  (1987,  March).  Job  performance  measurement  and  the  joint- 
service  project:  An  overview.  In  H.  G.  Baker  &  G.  J.  Laabs  (Eds.), 
Proceedings  of  the  Department  of  Defense/Educational  Testing 
Service  Conference  on  Job  Performance  Measurement  Technologies. 
San  Diego,  CA:  Navy  Personnel  Research  and  Development  Center. 

This  report  reflects  comments  made  by  the  author  during  the 
opening  ceremonies  of  the  Job  Performance  Measurement 
Technologies  Conference  held  in  March,  1987,  in  San  Diego,  California. 
Serving  as  Project  Officer  for  the  Joint-Service  Project,  the  author 
briefly  describes  the  history  of  job  performance  measurement 
activities  since  the  initiations  provided  by  Congress  in  1980.  The 
overall  management  structure  for  the  Joint-Service  Project  is 


presented  including  the  composition  and  basic  responsibilities  of  the 
Manpower  Accession  Policy  Steering  Committee,  the  Joint-Service  Job 
Performance  Measurement  Working  Group,  and  the  National 
Academy  of  Sciences  Committee  on  the  Performance  of  Military 
Personnel.  The  ultimate  goal  of  the  project  is  "to  validate  quality 
enlistment  standards  against  actual  job  performance,  instead  of 
success  in  training."  Furthermore,  once  the  research  establishes  a 
definitive  link  between  enlistment  criteria  and  job  performance,  "we 
will  then  be  able  to  determine  accession  quality  requirements  with 
greater  precision  than  is  now  possible." 

Hedge,  J.  W.,  &  Teachout,  M.  S.  (1986).  Job  performance  measurement:  A 
systematic  program  of  research  and  development  (AFHRL-TP-86-37, 
AD-A174  175^.  Brooks  AFB,  TX:  Training  Systems  Division,  Air  Force 
Human  Resources  Laboratory. 

This  paper  describes  the  Air  Force's  strategy  for  developing, 
evaluating,  and  selecting  a  measurement  methodology  for  use  in 
validating  selection  systems  and  evaluating  training  programs  across 
a  wide  range  of  job  specialties.  Background,  current  status,  and 
future  plans  of  the  job  performance  measurement  project  are 
discussed  in  terms  of  alleviating  the  criterion  problem  that  has 
defied  solution  by  industrial  psychologists  for  many  years  (authors’ 
abstract). 

Hendrix,  W.  H.,  Ward,  J.  H.,  Jr.,  Pina,  M.,  Jr.,  &  Haney,  D.  L.  (1979).  Pre- 
enlistment  person-job  match  system  (AFHRL-TR -79-29,  AD-A078 
427).  Brooks  AFB,  TX:  Occupation  and  Manpower  Research  Division, 
Air  Force  Human  Resources  Laboratory. 

Research  discussed  in  this  report  involved  the  development  of 
a  computer-based  assignment  system  which  became  operational  Air 
Force  wide  1  November  1976.  The  Person-Job  Match  System  was  the 
computer  algorithm  which  matched  potential  recruits  with  available 
jobs.  This  algorithm,  when  operationally  implemented,  became  the 
assignment  algorithm  of  the  Air  Force  Advanced  Personnel  Data 
System’s  Procurement  Management  Information  System  (APDS- 
PROMIS).  This  report  presents  the  assignment  concept,  methods  of 
combining  multiple  objectives,  the  conceptual  payoff  equation, 
baseline  performance  data,  and  future  directions  anticipated 
(authors'  abstract). 
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Kyllonen,  P.  C.,  &  Christal,  R.  E.  (1988).  Cognitive  modeling  of  learning 

abilities:  A  status  report  of  LAMP  (AFHRL-TP-87-66,  AD-A190  671). 
Brooks  AFB,  TX:  Manpower  and  Personnel  Division,  Air  Force  Human 
Resources  Laboratory. 

This  paper  describes  some  of  the  research  activities  underway 
as  part  of  the  Air  Force's  Learning  Abilities  Measurement  Program 
(LAMP).  A  major  objective  of  this  basic  research  project  is  to  devise 
new  models  of  the  nature  and  organization  of  human  abilities,  with 
the  long-term  goal  of  applying  these  models  to  improve  current 
personnel  selection  and  classification  systems.  The  activities  of  the 
project  have  been  divided  into  two  categories.  The  first  category  is 
concerned  with  identifying  fundamental  learning  abilities  by 
determining  how  learners  differ  in  their  abilities  to  think,  remember, 
solve  problems,  and  acquire  knowledge  and  skills.  From  research 
already  completed  ,  a  four-source  framework  has  been  established 
that  assumes  observed  learner  differences  to  be  due  to  differences  in 
processing  speed;  processing  capacity;  and  the  breadth,  extent,  and 
accessibility  of  conceptual  knowledge  and  procedural  skills.  The 
second  category  of  research  activities  is  concerned  with  validating 
new  models  of  learning  abilities.  To  do  this,  a  number  of  computer¬ 
ized  intelligent  tutoring  systems  have  been  built  that  serve  as  mini¬ 
courses  in  technical  areas  such  computer  programming  and  trouble¬ 
shooting  electrical  circuits.  A  major  objective  of  this  part  of  the 
program  is  to  develop  principles  for  producing  indicators  of  student 
learning  progress  and  achievement.  These  indicators  will  serve  as 
the  learning  outcome  measures  against  which  newly  developed 
learning  abilities  tests  will  be  evaluated  in  future  validation  studies 
(authors'  abstract). 

Maginnis,  E.  B.,  Uchima,  A.,  &  Smith,  C.  E.  (1975a).  Establishing  aptitude 
requirements  for  Air  Force  jobs:  Historical  review  of  aptitude  levels 
and  impact  on  the  personnel  system  (AFHRL-TR-75-44  [I],  AD-A023 
250).  Lackland  AFB,  TX:  Occupational  and  Manpower  Research 
Division,  Air  Force  Human  Resources  Laboratory. 

This  report  is  the  first  in  a  series  of  three.  It  presents  a  review 
of  the  military  service  aptitude  batteries,  with  some  comparisons  of 
minimum  aptitude  scores  required  for  entry  to  military  occupational 
areas.  Changes  in  Air  Force  aptitude  score  minimums  are  traced  and 
summarized.  A  systems  analysis  of  estimated  impacts  of  different 
aptitude  levels  for  enlistment  on  elements  of  the  personnel  system 
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leads  to  a  statement  of  salient  negative  influences  on  promotion, 
performance,  and  job/service  satisfaction  (authors’  abstract). 

Maginnis,  E.  B.,  Uchima,  A.,  &  Smith,  C.  E.  (1975b).  Establishing  aptitude 
requirements  for  Air  Force  jobs:  Some  personnel  system  actions,  to 
offset  negative  impacts  of  aptitude  changes  (AFHRL-TR-75-44  [II], 
.AD-A022  206).  Lackland  AFB,  TX:  Occupational  and  Manpower 
Research  Division,  Air  Force  Human  Resources  Laboratory. 

If  aptitude  level  for  entry  to  an  occupational  specialty  is  set  at 
higher  or  lower  than  optimal,  there  can  be  negative  impacts  on 
certain  elements  of  the  personnel  system.  This  report  considers 
personnel  system  actions  that  might  offset  these  negative  effects. 

Some  relationships  between  system  actions  and  Air  Force  needs  are 
presented  first.  Then  personnel  system  actions  with  potential  value 
are  identified  and  judged  for  their  impact  on  system  elements. 

Finally,  the  actions  so  identified  are  evaluated  for  potential  effect¬ 
iveness  in  offsetting  negative  impacts  under  conditions  of  lower,  and 
then  higher,  aptitude  than  the  optimal.  Some  relative  estimates  of 
cost  of  system  actions  are  given  as  examples  of  an  approach  to 
selecting  actions  for  implementation.  Of  the  five  alternative  actions 
evaluated,  the  most  preferred  was  one  that  provides  the  greatest 
opportunity  for  exposure  to  a  maximum  of  jobs  within  a  specialty 
(authors'  abstract). 

Maginnis,  E.  B.,  Uchima,  A.,  &  Smith,  C.  E.  (1975c).  Establishing  aptitude 

requirements  for  Air  Force  lobs:  Methodological  approaches  (AFHRL- 
TR-75-44  [III],  .AD-A022  250).  Lackland  AFB,  TX:  Occupational  and 
Manpower  Research  Division,  Air  Force  Human  Resources  Laboratory. 

This  report  presents,  using  a  system-oriented  approach, 
alternative  methodologies  that  could  be  used  to  establish  aptitude 
requirements  for  Air  Force  occupations.  It  covers  a  description  of 
the  aptitude  requirements  system,  a  review  of  the  interaction  of 
aptitude  requirements  and  personnel  system  actions,  and  a  func¬ 
tional  now  for  the  requirements  system.  A  flow  for  the  develop¬ 
mental  activities  necessary  to  design  and  implement  the  system  is 
also  given,  followed  by  recommendations  for  development  of  a 
methodology  for  determining  aptitude  requirements  for  effective  job 
performance,  with  separate  consideration  for  optimal  aptitude  types 
and  levels  with  respect  to  career  development,  assignment  flexibility, 
and  job  satisfaction  (authors'  abstract). 
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May,  L.  J.  (1986).  Educational  quality  requirements  for  Marine  Corps 
enlisted  personnel  (CNR  121).  Alexandria,  VA:  Center  for  Naval 
Analyses. 

This  report  sets  forth  a  methodology  for  determining  the 
optimal  mix  of  Marine  Corps  enlisted  personnel.  Assuming  a  goal  of 
maximizing  net  benefit,  high  school  graduates  and  nongraduates 
were  evaluated  in  terms  of  both  cost  and  performance  differences. 
High  school  graduates  cost  more  to  recruit  than  nongraduates  but 
have  a  lower  attrition  rate.  In  addition,  there  is  abundant  evidence 
that  high  school  graduates  perform  better  than  nongraduates  on  the 
job.  Educational  requirements  for  new  accessions  were  determined 
for  several  cost  and  relative-performance  scenarios  (author's 
abstract). 

May,  L.  J.  (1987).  Alternative  modeling  approaches  for  setting  cost- 
effective  qualification  standards  (CRM  87-54).  Alexandria,  VA: 

Center  for  Naval  Analyses. 

The  Rand  Corporation  has  developed  a  model  that  determines 
cost-effective  enlistment  standards  for  military  occupational  spec¬ 
ialties,  Since  this  model  may  have  an  impact  on  enlistment  policy,  it 
is  important  that  the  modeling  approach  used  to  determine  the 
enlistment  standards  be  fully  understood.  This  research  memoran¬ 
dum  describes  the  modeling  technique  used  by  the  Rand  Corporation 
and  compares  the  Rand  approach  to  alternative  modeling  methods  in 
an  attempt  to  determine  the  sensitivity  of  the  model  outcome  to  the 
methodology  employed  (author's  abstract). 

May,  L.  J.,  &  Mayberry,  P.  W.  (1986).  The  Rand  cost-performance  model 

for  setting  qualification  standards:  preliminary  comments  (CRM  86- 
200).  Alexandria,  VA:  Center  for  Naval  Analyses. 

The  Rand  Corporation  has  developed  a  model  to  determine 
cost-effective  qualification  standards  for  military  occupational 
specialties.  Since  this  model  may  potentially  have  an  impact  on 
enlistment  policy,  it  is  important  to  fully  understand  its  assumptions 
and  procedures.  This  research  memorandum  examines  key  elements 
of  the  model,  discusses  the  impact  they  have  on  the  outcome  of  the 
model,  and  identifies  issues  that  might  benefit  from  further  analysis 
(authors'  abstract). 
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Mayberry,  P.  W.  (1986).  From  hands-on  measurement  to  iob 
performance:  The  issue  of  generalizabilitv  (CRM  86-214). 

Alexandria,  VA:  Center  for  Naval  Analyses. 

The  generalization  from  hands-on  test  scores  to  performance  in 
a  military  occupational  specialty  can  be  threatened  by  many  potent¬ 
ial  sources  of  error  within  the  measurement  process.  Such  sources  of 
error  can  include  scoring  inconsistencies  by  test  administrators, 
testing  over  a  long  period,  and  diverse  test  content.  This  analysis 
estimates  the  influence  of  these  factors  on  the  hands-on  scores  for 
three  Marine  Corps  specialties.  Estimates  of  test  reliability  are 
discussed  in  light  of  the  effect  of  the  measurement  factors  on  the 
hands-on  scores.  Research  designs  to  assess  specific  issues  of  relia¬ 
bility  are  proposed  for  the  full-scale  administration  of  hands-on  tests 
to  the  Infantry  occupational  field  (author's  abstract). 

Mumford,  M.  D.,  Weeks,  J.  L.,  Harding,  F.  D.,  &  Fleishman,  E.  A.  (1987). 
Measuring  occupational  difficulty:  A  construct  validation  against 
training  criteria.  Journal  of  Applied  Psychology.  72(4).  578-587. 

Burtch,  Lipscomb,  and  Wissman's  (1982)  occupational  learning 
difficulty  index  attempts  to  measure  the  difficulty  of  occupations  by 
aggregating  workers'  evaluations  of  task  learning  time.  In  the 
present  study  we  examined  the  construct  validity  of  this  job  analysis 
index.  To  accomplish  this,  48  different  occupational  training  pro¬ 
grams  were  described  in  terms  of  15  training  content  variables,  6 
student  characteristics  variables,  and  7  training  performance  vari¬ 
ables.  The  results,  obtained  in  a  correlational  analysis,  indicated  that 
the  occupational  learning  difficulty  index  yielded  an  interpretable 
pattern  of  relationships  with  the  training  content  and  performance 
variables.  We  conclude  that  this  task  learning  time  index  displays 
some  construct  validity  as  a  measure  of  occupational  difficulty  and, 
therefore,  should  prove  of  value  in  designing  training,  manpower 
allocation,  and  job  evaluation  systems  (authors'  abstract). 

National  Academy  of  Sciences  (1987).  Summary  report:  Competency 
measurement  subgroup  meeting.  October  24-25,  1986  and  March 
13-14,  1987.  (Unpublished  Manuscript.) 

This  report  documents  the  discussions  between  the  National 
Academy  of  Sciences  (  .NAS  )  Committee  on  the  Performance  of 
Military  Personnel  and  the  Joint-Service  Job  Performance 
Measurement  (  JPM  )  Working  Group  concerning  the  NAS 
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Committee's  recommendation  to  focus  job  performance  measure¬ 
ment  research  on  the  development  of  competency-based  measures. 
That  is  ,  the  research  design  should  focus  on  having  an  absolute  scale 
of  performance,  with  scores  referenced  not  to  the  relative  scores  of 
others  but  to  levels  of  job  mastery.  Rationale  for  the  recommenda¬ 
tion  is  presented  along  with  other  proposed  uses  of  job  p.'rformance 
measures. 

Nord,  R.  ,  &  White,  L.  (1987).  The  measurement  and  application  of 
performance  utility:  Some  kev  issues.  Paper  prepared  for  the 
Committee  on  the  Performance  of  Military  Personnel,  Commission  on 
Behavioral  and  Social  Sciences  and  Education,  National  Research 
Council,  Washington,  D.  C. 

A  primary  concern  of  this  paper,  and  of  the  research  it 
describes,  is  with  the  complex  relationship  between  the  value  of  a 
soldier's  performance  in  a  job  and  the  value  of  a  distribution  of 
soldier  performance  within  and  across  jobs.  In  particular,  we 
examine  the  issue  of  how  "  performance  value"  in  a  job  changes  as  a 
function  of  changes  in  the  distribution.  The  importance  of  this 
variation  depends  on  several  things,  including  (a)  the  nature  of  the 
alternatives  being  evaluated;  (b)  the  degree  to  which  individual 
performance  is  likely  to  be  similar  in  different  jobs  (the  dimension¬ 
ality  of  performance);  and  (c)  the  magnitude  of  the  differences  in 
distributions  that  can  result  by  choosing  different  policies. 

The  questions  addressed  in  the  paper  are  exclusively  appli¬ 
cable  to  measurement  of  performance  value  in  a  context  where 
conventional  economic  theory  may  appear  inapplicable.  However,  the 
paradigms  embedded  in  micro-economic  production  theory  provide  a 
convenient  and  useful  framework  within  which  the  assumptions 
embodied  in  alternative  measurement  approaches  can  be  examined. 
Within  this  framework,  soldier  performance  in  a  given  job  can  be 
treated  as  a  "factor"  or  input  to  a  hierarchical  production  process,  and 
the  "value"  of  performance  is  defined  in  terms  of  its  immediate 
contribution  to  job  output  and  indirectly  through  the  contribution  of 
the  job  to  the  Army's  mission(s)  (authors'  introduction)  . 


Office  of  the  Assistant  Secretary  of  Defense,  Manpower,  Installations,  and 
Logistics  (1984).  Joint-service  efforts  to  link  enlistment  standards  to 
job  performance.  Third  annual  report  to  the  House  Committee  on 
Appropriations.  U.  S.  Department  of  Defense,  Washington,  D.  C. 


See  OASD  (1986). 

Office  of  the  Assistant  Secretary  of  Defense,  Force  Management  and 

Personnel  (1985).  Joint-service  efforts  to  link  enlistment  standards 
to  job  performance.  Fourth  annual  report  to  the  House  Committee  on 
Appropriations.  U.  S.  Department  of  Defense,  Vv^shington,  D.  C. 

See  OASD  (1986). 

Office  of  the  Assistant  Secretary  of  Defense,  Force  Management  and 

Personnel  (1986).  Joint-service  efforts  to  link  enlistment  standards 

to  job  performance. _ FJfth  .annual  report  to  the  House  Committee  on 

Appropriations.  U.  S.  Department  of  Defense,  Washington,  D.  C. 

This  report,  which  is  submitted  annually  to  the  House  and 
Senate  Committees  on  Armed  Services  and  Appropriations,  sum¬ 
marizes  the  history  of  the  Joint-Service  Job  Performance  Measure¬ 
ment/Enlistment  Standards  Project-  The  authors  (OASD,  1986;  ii-iii) 
provide  the  following  overview: 

This  fifth  annual  report  builds  on  the  first  four,  which 
focused  on  the  strategy,  structure,  and  implementation  of  the 
DoD  effort.  The  first  report  (December  1982)  outlined  the 
actions  taken  to  establish  formal  Office  of  the  Secretary  of 
Defense  oversight  of  the  Joint-Service  Project.  The  second 
report  (December  1983)  described  the  Joint-Service  research 
strategy  for  developing  job  performance  measures  and  testing 
the  feasibility  of  linking  enlistment  standards  to  job  perform¬ 
ance  data.  The  third  and  fourth  reports  (December  1984  and 
1985)  described  the  Services'  progress  in  implementing  the 
overall  research  strategy.  This  report:  1)  highlights  the 
significant  issues,  events,  and  accomplishments  during  calendar 
year  (CY)  1986;  2)  reviews  the  Service-by-Service  progress  in 

developing  and  implementing  job  performance  measures;  and 
3)  outlines  the  National  Academy  of  Sciences  Committee  on  the 
Performance  of  Military  Personnel's  recommendations  and 
evaluation  of  the  Joint-Service  Project. 

Specifically,  the  contents  of  this  report  are  as  follows: 
Chapter  One  summarizes  the  Project's  history  and  working 
relationships  with  oversight  groups.  Chapter  Two  outlines  the 
Project's  research  and  development  strategy.  Chapter  Three 
describes  significant  events  and  accomplishments  of  the  Joint- 
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Service  effort  during  CY  1986.  Chapter  Four  contains  the 
National  Academy  of  Sciences’  recommendations  for  the  Joint- 
Service  Project.  Chapters  Five  through  Eight  report  individual 
Service  progress. 

Schmitz,  E.  J.  (1987).  Improving  personnel  performance  through 
assignment  policy.  Paper  prepared  for  the  Committee  on  the 
Performance  of  Military  Personnel,  Commission  on  Behavioral  and 
Social  Sciences  and  Education,  National  Research  Council,  Washington, 
D.C. 


The  U  S  Army  Research  Institute  (ARI  )  has  developed  the 
Enlisted  Personnel  Allocation  System  (  EPAS  )  to  recommend  training 
assignments  to  new  recruits.  This  system,  based  upon  a  combination 
of  optimization  and  decision  rules,  can  increase  soldier  performance 
substantially  over  the  present  assignment  system.  Results  of  EPAS 
simulations  are  compared  with  the  present  system,  increased 
enlistment  standards,  and  improvement  of  the  job  composite  tests. 
Finally,  economic  and  psychological  models  are  used  to  assess  the 
value  of  these  performance  increases  to  the  Army  (author's  abstract). 

Sellman,  W.  S.  (1987,  March).  Response  to  panel  discussion  on  issues  in 
the  implementation  of  a  job  performance  measurement  program.  In 
H.  G.  Baker  &  G.  J.  Laabs  (Eds.),  Proceedings  of  the  Department  of 
Defense/Educational  Testing  Service  Conference  on  Job  Performance 
Measurement  Technologies.  San  Diego,  CA;  Navy  Personnel  Research 
and  Development  Center. 

On  the  last  day  of  the  Job  Performance  Measurement 
Technologies  Conference  held  in  March,  1987,  in  San  Diego,  California, 
a  panel  discussion  was  held  to  address  various  issues  in  the  imple¬ 
mentation  of  a  job  performance  measurement  program.  The  panel 
was  composed  of  five  members  of  the  Joint-Service  Job  Performance 
Measurement  Working  Group.  This  report  summarizes  three  obser¬ 
vations  made  by  Dr.  Sellman  as  Director  of  Accession  Policy,  .Office  of 
the  Assistant  Secretary  of  Defense,  Force  Manpower  and  Planning. 

The  first  observation  concerned  the  panel's  discussion  of  the  Army 
Skill  Qualification  Test  (SQT)  program.  Dr.  Sellman  points  out  one 
important  event  not  mentioned  by  the  panel  was  the  impact  of  the 
ASVAB  misnorming  on  the  SQT  program.  The  second  observation 
emphasized  Dr.  Sellman's  philosophy  against  the  concept  of  "if  it  ain't 
broke,  don't  fix  it."  The  third  observation  provided  clarification  that 
the  purpose  of  the  Job  Performance  Measurement  Project  is  "not  to 
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validate  ASVAB  against  job  performance.  The  purpose  is  to  validate 
enlistment  standards  against  job  performance." 

Strickland,  W.  (1987,  March).  Panel  discussion  on  issues  in  the  imple¬ 
mentation  of  a  job  performance  measurement  program.  In  H.  G. 

Baker  &  G.  J.  Laabs  (Eds.),  Proceedings  of  the  Department  of 
Defense/Educational  Testing  Service  Conference  on  Job  Performance 
Measurement  Technologies.  San  Diego,  CA:  Navy  Personnel  Research 
and  Development  Center. 

On  the  last  day  of  the  Job  Performance  Measurement 
Technologies  Conference  held  in  March,  1987,  in  San  Diego,  California, 
a  panel  discussion  was  held  to  address  various  issues  in  the  imple¬ 
mentation  of  a  job  performance  measurement  program.  The  panel 
was  composed  of  five  members  of  the  Joint-Service  Job  Performance 
Measurement  Working  Group.  This  report  contains  the  comments 
made  by  the  author  as  a  panel  member  representing  the  Accession 
and  Reenlistment  Policy  office  at  the  Air  Staff  level.  One  area  the 
author  focuses  attention  on  is,  beyond  validation  of  ASVAB,  what 
research  products  will  be  implemented? 

Ward,  J.  H.,  Jr.  (1977).  Creating  mathematical  models  of  judgment 

processes:  From  policv-capturing  to  policy  specifying  (  AFHRL-TR- 
77-47,  AD-A048  983).  Brooks  AFB  TX:  Occupation  and  Manpower 
Research  Division,  Air  Force  Human  Resources  Laboratory. 

Planning  for  a  computer-based  personnel  job  opportunities 
system  for  the  Air  Force  led  to  a  requirement  for  a  procedure  to 
generate  a  "payoff  or  "value"  of  the  assignment  of  each  person  to 
each  possible  job.  This  report  discusses  three  methods  of  weighting 
different  information  to  form  a  single  indicator  of  "payoff  or  "value," 
explicit  weighting  and  two  implicit  weighting  methods  --  policy¬ 
capturing  and  policy-specifying.  The  two  implicit  weighting  methods 
are  combined  into  a  more  comprehensive  method  referred  to  as 
policy-development. 

The  policy-capturing  process  presents  a  series  of  decision 
situations  to  one  (or  more)  policy  makers  and  the  policy  maker 
assigns  to  each  situation  a  number  which  reflects  the  "value"  or 
"payoff."  Then  a  mathematical  model  is  derived  by  obtaining  the 
regression  equation  that  best  predicts  the  policy  maker's  judgments. 
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Policy-specifying,  which  is  the  main  focus  in  this  report,  does 
not  depend  on  a  sample  of  actual  judgments  to  determine  the  re¬ 
gression  weights,  but  attempts  to  translate  into  mathematical  form  a 
policy  maker’s  more  global  statements  about  the  general  properties 
that  the  model  should  have.  The  mixing  of  policy-capturing  and 
policy-specifying  leads  to  a  process  called  policy-development 
(author's  abstract  ). 

Waters,  B.  K.,  Laurence,  J.  H.,  &  Camara,  W.  J.  (1987).  Personnel  enlistment 
and  classification  procedures  in  the  U.  S.  Military.  Paper  prepared 
for  the  Committee  on  the  Performance  of  Military  Personnel, 
Commission  on  Behavioral  and  Social  Sciences  and  Education,  National 
Research  Council,  Washington,  D.  C. 

This  paper  begins  its  analyses  of  the  selection  and  classification 
processes  within  the  military  by  first  examining  the  topics  in  a  more 
general  setting  outside  the  military.  Section  11  discusses  the  con¬ 
ceptual  basis  for  selection  decisions  with  regard  to  setting  standards, 
establishing  cut  scores  on  selection  tests,  and  validating  the  selection 
procedures.  It  ends  with  a  comparison  of  the  similarities  and 
differences  between  the  military  and  civilian  environments.  Sections 
III  and  IV  describe  the  military's  selection  and  classification 
systems,  respectively.  Section  V  focuses  on  the  processes  used  by 
the  Military  Services  to  set  and  revise  their  selection  and  classifica¬ 
tion  standards:  how  the  standards  evolved,  how  they  are  changed, 
and  the  rationale  for  changes.  Key  accession  and  recruiting  policy 
officials  were  interviewed  to  determine  who  recommends  and  who 
makes  the  decisions  of  Service  enlistment  selection  and  classification 
standards  and  upon  what  bases  the  decisions  are  made.  Finally,  the 
information  gathered  and  reported  is  used  in  Section  VI  to  develop  a 
model  of  military  selection  and  classification  processes.  Examples 
from  the  Military  Services  are  described.  The  paper  ends  with  a 
brief  discussion  of  conclusions  from  the  study  (author's  introduction). 

Weeks,  J.  (1984).  Occupational  learning  difficulty.  A  standard  for 

determining  the  order  of  aptitude  requirement  minimums  (AFHRL- 
SR-84-26,  AD-A147  410).  Brooks  AFB,  TX:  Manpower  and  Personnel 
Division,  Air  Force  Human  Resources  Laboratory. 


Entry  into  Air  Force  enlisted  job  specialties  is  largely  governed 
by  aptitude  requirements.  Specification  of  aptitude  requirements 
involves  two  separate  decisions:  an  aptitude  type  (i.  e.,  the 
Mechanical,  Electronics,  General,  or  Administrative  aptitude  index  of 


the  Armed  Services  Vocational  Aptitude  Battery)  is  identified;  then,  a 
minimum  level  of  aptitude  (i.  e.,  a  percentile  score  cutoff)  is  deter¬ 
mined.  Although  identification  of  the  appropriate  aptitude  type  is 
relatively  straightforward,  determination  of  the  aptitude  minimum  is 
complicated  by  several  considerations.  The  problem  which  served  as 
the  basis  of  this  research  was  the  need  for  a  quantitative  method  for 
determining  percentile  score  cutoffs. 

A  method  for  determining  aptitude  minimums  based  on 
occupational  survey  information  is  described.  For  job  specialties 
having  a  common  aptitude  requirement  type,  measures  of 
occupational  learning  difficulty  are  proposed  as  a  frame  of  reference 
for  determining  the  order  of  aptitude  minimums.  Rather  than 
representing  how  difficult  tasks  are  to  perform,  a  measure  of 
occupational  leai.iing  difficulty  for  a  given  specialty  represents  how 
much  ume  it  takes  to  learn  to  perform  associated  tasks.  It  is 
recommended  that  percentile  score  cutoffs  be  established  so  that  the 
order  of  the  cutoffs  corresponds  to  the  order  of  specialties  in  terms 
of  occupational  learning  difficulty.  Assuming  high-aptitude  enlistees 
learn  faster  than  low-aptitude  enlistees,  the  recommended  procedure 
will  ensure  that  job  specialties  which  require  the  most  time  to  learn 
are  manned  by  enlistees  who  learn  the  fastest. 

To  produce  measures  of  occupational  learning  difficulty, 
comprehensive  occupational  analyses  were  conducted  for  more  than 
200  job  specialties  and  over  10,000  job  types.  Comparisons  of  the 
order  of  aptitude  mimimums  (i.  e.,  percentile  score  cutoffs)  with  the 
order  of  specialties  in  terms  of  learning  difficulty  indicate  that 
aptitude  minimums  for  some  specialties  are  seriously  misaligned. 

Some  specialties  high  in  learning  difficulty  have  low  minimums,  and 
other  specialties  low  in  learning  difficulty  have  high  minimums. 

Such  misalignments  suggest  that  the  talent  available  to  the  Air  Force 
is  not  being  allocated  in  the  most  optimal  manner  (author's  abstract). 

Wigdor,  A.  K.,  &  Green,  B.  F.,  Jr.,  (Eds.).  (1986).  Assessing  the  performance 
of  enlisted  personnel.  Washington,  D.  C.:  National  Academy  Press. 

The  Committee  on  the  Performance  of  Military  Personnel, 
formed  within  the  National  Academy  of  Sciences  in  late  1982,  is 
composed  of  nationally  recognized  experts  in  scientific  and  technical 
areas  related  to  the  Joint-Service  Job  Performance  Measurement/ 
Enlistment  Standards  Project.  The  Committee's  purpose  is  to  provide 
independent  technical  review  of  the  Joint-Service  Project  research 


activities.  This  comprehensive  report  (121  pages)  documents  a 
recent  assessment  of  the  Project. 

The  report  is  organized  into  two  sections.  The  first  section 
focuses  on  work  accomplished  to  date  and  the  second  section  on 
upcoming  issues  in  analyzing  the  data  collected  with  the  newly 
developed  criterion  measures.  The  project  is  "in  the  process  of 
transforming  the  pivotal  issue  in  criterion  research  from  that  of 
demonstrating  the  validity  of  a  particular  measure  to  the  more 
complex  task  of  comparing  the  substantive  and  psychometric 
adequacy  of  alternative  criterion  measures." 

Wise,  L.  L.,  Campbell,  J.  P.,  &  Arabian,  J.  M.  (1987).  The  Armv  synthetic 
validation  project.  Paper  prepared  for  the  Committee  on  the 
Performance  of  Military  Personnel,  Commission  on  Behavioral  and 
Social  Sciences  and  Education,  National  Research  Council,  Washington, 
D.C. 

This  project  is  designed  "  to  identify  and  evaluate  alternative 
procedures  for  validating  potential  selection  composites  when  a  full 
empirical  validation  is  simply  not  feasible.  The  project  also  will 
examine  procedures  for  establishing  performance  standards  for  each 
job  and  using  these  standards  to  set  minimum  qualifying  scores  on 
new  selection  composites."  This  report  describes  the  synthetic 
validation  approach  including  critical  assumptions  and  details  of  the 
project  plan. 
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